This is a review of book that has helped me stretch my understanding of mutimodality. Although it is a 2003-publication, it outlines different approaches to multimodal texts and applications in educational contexts.
EDITORS: Carey Jewitt & Gunther Kress
TITLE: Multimodal Literacy
SERIES: New literacies and digital epistemologies v.4
PUBLISHER: Peter Lang
YEAR: 2003
Introduction
This collection of papers addresses multimodality from an educational perspective with an orientation on learning. Particularly, it focuses on how semiotic modes (image, gesture etc.) that structure all types of communication exchanges contribute to learning. The volume contains contributions by different authors that examine multimodality across a range of curriculum subjects in a variety of settings such as primary and secondary school classrooms and the home. The basic assumption of the book is that “meanings are made, distributed received, interpreted and remade in interpretation through many representational and communicative modes” (p. 1) not just through speech or written language. Thus, the papers here are based on a combination of a multimodal semiotic approach with other theoretical perspectives, illustrating research themes, achievements and theoretical tensions of this emergent area of inquiry.
The introduction by the editors, Carey Jewitt & Gunther Kress, “Multimodality new questions for learning” reviews the principal topics, concepts and methodologies addressed in the volume by the different contributors. Jewitt & Kress suggest that the traditional approach to communication and learning “is confused and contradictory” (p. 3) since it has focused on speech and writing, ignoring the multimodal nature of meaning making. From a multimodal perspective all semiotic modes partially contribute to the whole of meaning and communication. Learning in turn should be thought of as a multimodal process. The authors assert that the key concepts proposed in the multimodal social semiotic perspective are used differently in each of the chapters as reflected in the remainder of the collection.
Summary
Diane Mavers, “Communicating Meanings through Image Composition, Spatial Arrangement and Links in Primary School Students’ Maps” (Chapter 2), addresses students’ communication of meaning through their image-based mind maps on the topic of “Computers in my world” (p. 20). The author uses content analysis to examine the affordances of drawings by looking at image size, positioning, spacing, distancing, linking and the degree of detail in similar or different items on the page. Additionally, she uses interviews and a 15-minute writing task to further explore students “apparent meaning intention” (p. 20) in their mind-maps. Mavers argues that forms of communications privileged in schools center on written language, narrowing the potentialities of other semiotic modes such as drawing. She concludes that the selection, representation and arrangement of images in the children’s mind maps draw on “criterial attributes”, which refer to key defining features that permit the communication of meaning effectively and economically with consideration of the needs of the reader.
Carey Jewitt’s “Computer Mediated Learning: The Multimodal Construction of Mathematical Entities on Screen” (Chapter 3) focuses a computer programming system called Playground and its role in the construction of the mathematical concept of bounce. Jewitt considers that the choice of communicational modes to a great extent shapes what users can do with it, how meaning making can take place and determines the engagement of users with the program. She explores this assumption through the examination of three stages (written description, drawing representation, computer programming) followed by two 7-year old students in the design of a game in Playground. The author concludes that the affordances provided by Playground repositions the learner, taking him from being a re-producer of knowledge to being a producer of knowledge” (p. 52). The different representational and communicational resources that brings about the move from page to screen, generates transformations of pedagogy and knowledge.
In “Tiger’s Big Plan: Multimodality and the Moving Image” (Chapter 4), Andrew Burn and David Parker describe the experience of integration of speech, music and animated film in a moving image production. Drawing on the framework of semiotic production: Discourses, design, production and distribution, proposed by Kress & Van Leeuwen (2001), the authors analyze the social and technological process that entail the design of the film Anansi and the Firefly. Burn and Parker introduce the concept of Kineikonic, a term that defines the mode of moving image as an ensemble that comprises particular conventions afforded by the practices of filming and editing. The analysis undertaken by the authors’ shows that elementary school students make meaning through the use of the affordances of the kineikonic mode such as: grouping (articulation of different modes in groups: groups of gestures, groups of lights, groups of image and sound etc.); boundary (what determines the edges between groups) and conjunction (how to move between groups). One of the main conclusions indicates that although most of the elements of animation design were taught to children (narrative conventions, the music, the visual convention of the moving image and so forth), there were some other elements (e.g. inter-texts from TV superheroes) that they brought unconsciously and that could “be drawn more consciously into the design process” (p. 71) in future experiences.
Gemma Moss’ ”Putting the Text Back into Practice: Junior-Age Non-Fiction as Objects of Design” (chapter 5) illustrates the affordances an resistances that text design of junior- age non-fiction offer to understanding literacy events. The author argues that in order to go beyond the information retrieval model of reading, reading and meaning should be understood as the product of the interaction between social and cultural situated participants. A look from the multimodal perspective would give new insights into what is happening in a literacy event. Moss reviews three well known non-fiction texts: The Hamlyn Junior Encyclopedia, The Eyewitness Guides and the series How Would You Survive? in order to exemplify how page layout has changed. She establishes that the double spread design, a finite composite of words and images, has become the norm in many school texts. While not giving enough support and description about the methodology assumed in the study, the author draws on ethnography of reading to present a general analysis of some samples of interaction of children with non-fiction texts. She concludes that multimodality reconfigures the text by looking at it as more than writing language or image. It enlarges the meaning potential where the reader ·”re-makes the text, drawing on the possibilities each mode represents” (p. 85).
The study by Charmian Kenner: “Embodied Knowledges: Young Children’s Engagement with the Act of Writing” (chapter 6) has by its own account very limited previous research to go on, and so focuses for the most part on a single case analysis of six students. It describes Chinese, Arabic, Spanish and English bilingual children’s bodily and cognitive engagement in the process of learning to write. Additionally, the author explores how script learning as a multimodal process embodies knowledges that children interpret and transform. Kenner suggests that the teaching of handwriting embodies different knowledges that enhance cultural patterns such as spatiality and directionality. Wring intimately links cognitive understanding with visual and physical apprehension in a multimodal arrangement. A variety of methods of data collection are described comprising: observations in school and in the children’s homes, as well as interviews and children’s peer teaching sessions. The chapter ends by giving a comprehensive analysis of the several dimensions concerning the act of writing in the five languages studied. It describes the type of writing process (analytic, synthetic); spatiality (centered, linear); directionality of text (vertical, horizontal); directionality of line (top down, right left, left right etc.) and directionality (symbol -left right top-bottom etc.). It concludes by stating that children observed in classes and peer teaching sessions showed awareness of the principles of visual organization of the page and its multiple affordances.
Chapter 7 by Lesly Lancaster: “Beginning at the Beginning: How a Young Child Constructs Time Multimodality”, throws light on the issue of how a 23-month old child –Anna- uses physical and bodily resources to construct the concept of time. Lancaster contends that time is socially and materially constructed due to the fact that our experience with it is relative to our experience of everyday events and embedded in a specific culture and community. Using the methodology of case study, the author explores, in particular, the child’s understanding of when events begin and finish. She analyses the use of the child’s semiotic resources “to structure the starting and completion of the making of a card for her mother” (p.110). To do this the author draws on ethnographic evidence, a multimodal analysis of language, vocalization, gesture and action and a hermeneutic explanation of the semiotic event based on Geertz’s (1983) concept of ‘thick description’. Findings indicate that the opening episode of the semiotic event in which Anna’s father arranges the materials to make the card, prompt her to rely on the mode of gaze. The use of gaze has a predictive function since Anna is able to structure the kind of activity to come and its beginning based on the objects (ink pot, sheet of paper) she sees and activities she associates with them. Her beginning to draw is mediated by physical and bodily modes (she leans across the table to be closer to have physical contact with the page), spatial mode (where to put the first mark on the paper), gaze and language mode (she speaks to her father). The eye move across the page and her putting the pen beside the paper signifies the potential completion of the activity and her satisfaction at the textual and affective level. The chapter concludes by pointing out that even at this young age children make use of a range of modalities to construct and represent abstract and symbolic forms such parameters of the beginning and ending of an event.
In “The Olifantsvlei Stories Project: Multimodality, Creativity and Fixing in the Semiotic Chain” (Chapter 8), Pippa Stein describes the literacy experience of a group of primary school students and teachers in the process of making doll/child figures to represent narratives. By comparing traditional fertility doll/child figures of the Southern African region, where this study took place, Stein asserts that “the children’s doll figures form part of the ongoing semiotic chain of social, cultural and aesthetic practices” (p.123) which have existed in this part of Africa for hundreds of years. However, the analysis of the semiotic resources used by children to make their dolls: materials (cloth, grass, reeds, wood, bottles etc.), shapes (conical), weight and height, exterior attachments (heads, arms and breasts) and the examination of an interview where they stated their motivation for the design of the dolls, showed that the remaking of these semiotic products was not “reproductive but innovative and transformative” (p. 133. Particularly, the way dolls were made extended the ‘grammar’ of doll’s making as cultural practice and reframed aspects related to how identity is built. For instance, while doll making is usually associated with women, this experience allowed boys to participate and interact with female members of their homes in order to build their dolls. The last part of the chapter shows some examples of the stories that children wrote. The written production materialized the last stage of the semiotic chain that in the context of this project began with the drawing and writing about the character and then the making of the character in a 3D figure. Stein concludes that the ‘Olifantsvlei Stories Project’ allowed children to represent, transform and recontextualize cultural and historical practices, throwing light into the “relations between creativity, multimodal pedagogies and resources for representation and learning” (p. 137).
Kate Paul’s chapter 9: “Children’s Text Making at Home: Transforming Meaning Across Modes” details an ethnographic case study carried out over a period of two years with Sam, a six-year old. The study focused on how meaning was constructed through multimodal texts in the home. Using field notes, photographs and transcribed oral narratives, the author illustrates how Sam built his sense of identity by transforming the cultural resources around him through play, the taking of photographs, the use of props and toy artifacts as well as the making of new ones. For example, Paul demonstrates the different semiotic modes (drawing, modeling, photograph, written and oral language) in which Sam transformed Pokemon characters (a Japanese card-game) to represent his meanings and his own identity. The detailed and analytic approach to Sam’s room and the general semiotic arrangement of the household led the author to conclude that cultural resources such as places: visited museums, Parmjit (Sam’ mom) work in schools, design materials, television and videos games constituted the main sources of Sam’s meaning making processes. The move across different modes as well as the recontextualization of Pokemon’s characters and the use of different material resources enacted different representations and constitution of a variety of signs. The author concludes that “the finished product was hybrid and adapted to the identity of the maker” (p. 153). She points at the need for school to take the challenge of supporting children’s creativity through multimodal learning environments.
Chapter 10 “Palmers’ Kiss: Shakespeare, School Drama and Semiotics” by Anton Franks, describes the experience of a teacher and a group of students (14-15 year-olds) in an inner-city school in London, who are preparing the play Romeo and Juliet. The main purpose of this paper is to examine the phases of a lesson in which a teacher and her students engage in the process of sedentary dramatized reading towards enactment of the text through different semiotic modes. Franks reports four stages developed in the class, mainly: opening stage when the teacher sets the stage for the activity, sitting in groups of five students around tables to work on excerpts of the play, practicing acting out the play-text, and finally showing each other’s scenes. Drawing on the concepts of mode and frame from multimodal semiotics, the author primary focuses on the second and third stages of the class. Through a detail analysis of the video-taped English lesson she finds that modes of bodily communication (gaze, posture, space and spatial positioning) are orchestrated to realize dramatic meaning in relation to author and audience. Although it is stated that the ways in which meaning is made in drama is very close to the ways meaning is made in everyday social life, findings indicate that the shifts between the two are difficult to identity due to their extensive overlap. The chapter ends by pointing at the need to carry out more work on how classroom actions are framed and what are the boundaries of the shifts between social and dramatic action.
The last chapter of this volume “Genres and the Multimodal production of Scientificness’ written by one of the pioneers in the study of multimodality: Gunther Kress, sheds light on representation of scientific meanings. It explores two questions: “how is knowledge reconfigured when ‘it’ is moved from one mode to another… and what can we learn about learning in the process of sign-making, and the reading of signs? (p. 172). Additionally the author sets out to discuss the category of genre in light of the combination of text modalities found in the samples that are object of examination. Two samples of texts produced in a science classroom by two students of an inner-city London school constitute the data of this analytical experience. The texts selected comprise a written report and a visual representation of the observation of cells in a piece of epidermis of an onion. Through a thorough description of the multimodal social semiotic process of the two texts, Kress explains that the two reports use language to position the sign makers in different social relations with what scientificness means. The first text relied on the genre structure of a recount and the second one on the conventions of a scientific procedure. Regarding the drawings, the first one presents a generalization of what was observed with a degree of abstraction whereas the second one presents the entity as it would appear to the human eye. Therefore, scientificness is carried out differently in the two cases. It is asserted that the genre of these two texts is of a mixed nature for which there is not label. This is explained by the fact that the sign maker is positioned somewhere between the everyday and the scientific world through the ensemble of verbal and visual modes. The author remarks that the way science is being represented in schools is in part owed to the central role that the image has acquired in books and the designs of screens in new media. Consequently a new theory of text is needed.
GENERAL EVALAUTION
In their introduction, the editors mention that the chapters in the book address multimodality and learning in a variety of contexts and across several subjects of the curriculum. These features appear as some of the best assets of this publication. The book is an insightful contribution to those who are interested in exploring the different approaches to research in the areas of focus: multimodality, social semiotics and the concept of mode in particular. As described in the introduction, the chapters of the volume find a common thread in the variety of ways in which the concept of semiotic mode is addressed in different contexts and through different methodologies, mainly: ethnography and case study.
The various chapters of the book manage to combine insights from the literature to inform their interpretations and analyses with, in most cases, detailed, relevant and revealing findings. Except for chapter 5, the rest of the chapters enable data to take center stage, therefore the authors treat and display data as the primary source of knowledge, and as the basis for their claims and interpretations. Additionally the volume offers useful examples of multimodal application with different age populations in schools. Although in almost all the papers the focus is Europe (only one article refers to an African context), particularly schools in London, many of the readings would be comparable in other contexts.
Yet there are some minimal issues that I would like to point out in regards to particular chapters of the volume. Firstly, I will concentrate of Maver’s article which offers a thorough description and analysis of the ‘criterial attributes’ of children’s mind maps. The paper, though, presents two main weaknesses in regards to the design of the study. First, the author reports that there was a nine-month gap between the drawing/writing and interviewing data collection procedures. As a reader one might wonder how children may have changed their understanding of computers and the interpretation of their drawings over a nine-month period. One implication is that explanations provided by them about the mind maps may not have corresponded to their mindset when they drew the drawings. A second shortcoming in connection to the interview regards its validity in the context of the study. According to Maver out of the three participants two were not interviewed which consequently affects the reliability of some of the conclusions presented in reference to the three participants. Although, the situations described above may appear normal under certain research conditions, the paper would have benefited from a discussion that addressed implications and limitations of the study.
Chapter 4 by Burn undertakes the description of moving image and in order to achieve this, he adopts the analytical framework proposed by Kress & Van Leeuwen (2001) in connection to the strata of semiotic production: discourse, design, production and distribution. Although, throughout the paper these elements are mentioned they did not reach in depth integration into the analyses carried out and as it would have been expected by a reader. The paper would have benefited from a more consistent and salient treatment of this term as regards the different process of moving image production. This could help readers who are new to Kress & Van Leeuwen’s proposal to have a better idea of the application of their concepts.
A final criticism goes to chapter 5 by Moss. Given the nature of the rest of papers in the volume I find this chapter slightly disappointing regarding the balance between theory and multimodal analysis. Moss expends a big portion of the article in the theoretical description of the different changes of layout in different non-fiction texts, nevertheless she never provides any visual image sample that could give a better sense of what she intended to convey. It is not clear to what extent this article can be considered as a report on a study; in fact the author does not clearly specify its nature. Although given the fact that the she mentions that the data presented belonged to a project she had been involved, readers might consider the article a research report. The author presents some samples of data which are not treated with depth of analysis. The discussion and presentation in more detail of the methodology used in the study and data samples in relation to multimodality are missing from the paper. The extended attention dedicated to the description of nonfiction texts does not contribute to enlarge the knowledge about the topic since authors like Kress and van Leeuwen have broadly dealt with the topic.
Aside from these minimal issues just addressed, the volume offers useful food for thought for researchers in different areas of the curriculum and from different disciplines that comprise linguistics, semiotics, anthropology and sociology among others. Overall, this book is a remarkable scholarly achievement which is indispensable for beginning researchers in the areas of social semiotics and multimodality in school contexts.