Learning Through Multimedia Interaction: The Construal of Primary Social Science Knowledge in Web-based Digital Learning Materials

Abstract

This thesis is concerned with the construal and the recontextualisation of primary social science knowledge in hypermedia texts. More specifically, it provides an account for the relations between verbiage and image in web-based multimodal interactive leaning materials, known as Multimodal Interactives (MIs). Based on the linguistic description, the thesis offers insights into the ways in which knowledge is construed and recontextaulised in the emerging electronic multimodal discourses. The general theoretical orientation of this thesis is that of systemic functional multimodal discourse analysis (SF-MDA). Within the framework of SF-MDA, the thesis proposes a complementary perspective on intersemiosis, which treats relations between verbiage and image as patterns formed during the unfolding of a text. To capture this type of intersemiotic relations, the thesis develops a logogenetic model for SF-MDA. The defining feature the model is the temporal axis (time), which serves as the main reference point for determining semiotic units (logogenetic units) and describing semiotic patterns (logogenetic patterns). The logogenetic model is applied in studying five MIs. The basic logogenetic unit used in analysis is Critical Path, the shortest traversal through a MI. Two types of logogenetic patterns along the Critical Paths in the five MIs are examined in detail, including intersemiotic ideational coupling and clustering. There are five basic types of verbiage-imaged coupling emerged from the analysis, including Naming & Identifying, Representing, Classifying & Co-classifying, and Circumstantiating. The analysis of ideational clustering shows the different ways in which participants and activities form clusters in each MI. By analysing intersemiotic coupling and clustering, the thesis shows that language and image construe the keys notions of primary social science such as people, place and community through three fundamental principles—abstraction, generalisation and specification. The study also demonstrates the possibility of achieving different degrees of pedagogic framing in hypermedia environments.