81̕ 22+811.1/.8

10.37482/2687-1505-V362

Natalya V. Dutova
e-mail: dutova_natalya@mail.ru, ORCID: https://orcid.org/0009-0003-4809-6865

The article provides a brief overview of the origination of the foreign socio-semiotic approach to multimodal discourse analysis. Using the methods of diachronic analysis, interdisciplinary synthesis, abstraction and analogy, the author describes the formation and development of complex socio-communicative research technologies in combination with socially constructed cultural resources. Primary attention is paid to the development of multimodal research in social semiotics. The article characterizes M. Halliday’s idea of metafunctionality in relation to language and human social life, which paved the way for the study of the joint functioning of semiotic resources in various discourses to achieve communicative goals. The theory of R. Hodge and G. Kress, which accorded a semiotic status to everything related to humans and their activity, is described. In addition, G. Kress and T. van Leeuwen’s contextual approach to investigating multimodality and M. O’Toole’s visual grammar are presented as universal technologies for the study of semiotic resources. Some features of contextual and grammatical approaches within the framework of systemic functional multimodal discourse analysis are outlined. Particular emphasis is given to the description of R. Scollon’s, S. Wong Scollon’s and S. Norris’ multimodal interactive analysis. The importance of R. Scollon’s idea about discourse as a dynamic social practice and language as a tool for performing actions is highlighted. S. Norris’ attempt to establish a universal unit of multimodal analysis as the smallest unit of pragmatic analysis in communication involving any modality is presented. Visual transcription is described as an effective method of analysing communication mediated by modern digital technologies.

multimodality, multimodal discourse, social semiotics, socio-semiotic approach, multimodal interactive analysis

1.  Kress G., Bezemer J. Multimodal Discourse Analysis. Handford M., Gee J.P. (eds.). The Routledge Handbook of Discourse Analysis. London, 2023, pp. 139–155. https://doi.org/10.4324/9781003035244-12
2.  O’Halloran K.L., Tan S. Discourse Analysis and Semiotics. Pelkey J., Cobley P. (eds.). Bloomsbury Semiotics. Vol. 4: Semiotic Movements. London, 2023, pp. 231–252. https://doi.org/10.5040/9781350139435.ch-11
3.  Norris S. Multimodal Theory and Methodology: For the Analysis of (Inter)action and Identity. London, 2020. 120 p.
4.  Kiose M.I. (comp.). Polimodal’nye izmereniya diskursa [Multimodal Dimensions of Discourse]. Moscow, 2022. 448 p.
5.  O’Halloran K.L. Multimodal Discourse Analysis. Hyland K., Paltridge B., Wong L.L.C. (eds.). The Bloomsbury Handbook of Discourse Analysis. London, 2021, pp. 249–267.
6.  Halliday M.A.K. Language as Social Semiotic: The Social Interpretation of Language and Meaning. London, 1978. 256 p.
7.  Hodge R., Kress G. Social Semiotics. Cambridge, 1988. 285 p.
8.  Kress G., van Leeuwen T. Reading Images: The Grammar of Visual Design. Abingdon, 2021. 288 р.
9.  O’Toole M. The Language of Displayed Art. Rutherford, 1994. 295 p.
10.  O’Halloran K.L., Beezer R.A., Farmer D.W. A New Generation of Mathematics Textbook Research and Development. ZDM Math. Educ., 2018, vol. 50, pp. 863–879. https://doi.org/10.1007/s11858-018-0959-8
11.  Djonov Е., Tseng C.-I. Harnessing the Potential of Transmedia Narratives for Critical Multimodal Literacy. Crit. Discourse Stud., 2021, vol. 18, no. 3, pp. 349–367. https://doi.org/10.1080/17405904.2020.1804954
12.  Caple H., Huan C., Bednarek M. Multimodal News Analysis Across Cultures. Cambridge, 2020. https://doi.org/10.1017/9781108886048
13.  Unsworth L. High School Science Infographics: Multimodal Meaning Complexes in Composite Image-Language Ensembles. Pensam. Educ., 2021, vol. 58, no. 2. https://doi.org/10.7764/PEL.58.2.2021.9
14.  Lemke J.L. Multiplying Meaning: Visual and Verbal Semiotics in Scientific Text. Martin J.R., Veel R. (eds.). Reading Science: Critical and Functional Perspectives on Discourses of Science. London, 1998, pp. 87–113.
15.  Jewitt C. Multimodality and Literacy in School Classrooms. Rev. Res. Educ., 2008, vol. 32, no. 1, pp. 241–267. https://doi.org/10.3102/0091732x07310586
16.  Wang Y. On Visual Semantic Algebra (VSA): A Denotational Mathematical Structure for Modeling and Manipulating Visual Objects and Patterns. IJSSCI, 2009, vol. 1, no. 4, pp. 1–16. https://doi.org/10.4018/jssci.2009062501
17.  Pierce C.S. Peirce on Signs: Writings on Semiotic by Charles Sanders Peirce. Chapel Hill, 1991. 294 p.
18.  Scollon R., Wong Scollon S. Discourses in Place: Language in the Material World. London, 2003. 258 p.
19.  Wertsch J.V. Voices of the Mind: Sociocultural Approach to Mediated Action. Cambridge, 1991. 182 p.
20.  Scollon R. Mediated Discourse: The Nexus of Practice. London, 2001. 182 p.
21.  Wong Scollon S., de Saint-Georges I. Mediated Discourse Analysis. Handford M., Gee J.P. (eds.). The Routledge Handbook of Discourse Analysis. London, 2011. Available at: https://www.researchgate.net/publication/307575422_Mediated_Discourse_Analysis (accessed: 14 April 2024).
22.  McNeill D. Hand and Mind: What Gestures Reveal About Thought. Chicago, 1992. 416 p.
23.  Geenen J. Introduction: Multimodal (Inter)action Analysis. Multimodal Commun., 2023, vol. 12, no. 1, pp. 1–6. https://doi.org/10.1515/mc-2023-0010
24.  Norris S. What Is a Mode? Smell, Olfactory Perception, and the Notion of Mode in Multimodal Mediated Theory. Multimodal Commun., 2013, vol. 2, no. 2, pp. 155–170. https://doi.org/10.1515/mc-2013-0008
25.  Norris S. Systematically Working with Multimodal Data: Research Methods in Multimodal Discourse Analysis. Hoboken, 2019. 301 р.
26.  Geenen J., Matelau-Doherty T., Norris S. Visual Transcription. A Method to Analyze the Visual and Visualize the Audible in Interaction. Rev. fr. méthodes visuelles, 2021, vol. 5, pp. 1–55.
27.  Jones R.H. Towards an Embodied Visual Semiotics: Negotiating the Right to Look. Thurlow C., Dürscheid C., Diémoz F. (eds.). Visualizing Digital Discourse: Interactional, Institutional and Ideological Perspectives. Berlin, 2020, pp. 19–45.
28.  Schmidt A., Marx K. Making Let’s Plays Watchable: An Interactional Approach to Gaming Visualizations. Thurlow C., Dürscheid C., Diémoz F. (eds.). Visualizing Digital Discourse: Interactional, Institutional and Ideological Perspectives. Berlin, 2020, pp. 131–151.
29.  Cserző D. Intimacy at a Distance: Multimodal Meaning Making in Video Chat Tours. Thurlow C., Dürscheid C., Diémoz F. (eds.). Visualizing Digital Discourse: Interactional, Institutional and Ideological Perspectives. Berlin, 2020, pp. 151–171.