Designing and Testing an AI System to Understand Student Muddy Card Responses

Abstract

Muddy cards are an active learning technique where students record the most confusing point from a lecture. They improve students’ memory retention and self-efficacy, and aid in developing their under- standing of course material. Despite their many benefits, reading and analysing muddy card responses requires a substantial time commitment from teachers. In this thesis, we developed a system that stream- lines the process of collecting and analysing muddy cards. This system uses embedding algorithms and agglomerative clustering to enable teachers to quickly identify the most commonly occurring confusing points in their lecture. We have also implemented a ‘student-assisted’ approach, where students identify peer responses that are semantically similar to what they entered to improve the quality of clustering. To explore the performance of the embedding models central to the muddy card system, we man- ually clustered 2,327 muddy cards (split into eight samples). We developed a new clustering metric called the ‘student questions answered satisfaction’ (SQAS) score to better measure clustering quality for muddy card applications. Using the SQAS score, we found that no embedding model markedly out- performed the others. This is in contrast to embedding benchmarks, which do reveal differences between embedding models. When incorporating the ‘student-assisted’ data in clustering, the ‘multi-evidence’ approach consistently improves SQAS score performance. To investigate the effectiveness of our muddy card system, we conducted a user study with 20 units at the University of Sydney. On a two-week cycle, teachers would use our clustering interface, and as a control, a baseline version with simple options for alphabetically sorting responses. Students and teachers completed an end-of-study survey, and 13 teachers agreed to be interviewed regarding their ex- perience with the system. During the study, there was a low student muddy card response rate. Through interviews, teachers explained that students do not generally complete optional activities, especially if they do not see the benefit in participating. We found that lecturers tended to prefer the clustering interface over the baseline interface. To build upon this study’s findings, we propose a follow-up user study where the muddy card system is modified to be more conducive to live lecture analysis. We hypothesise that this will increase the low response rate. The follow-up study will also include international universities to investigate the effect of societal differences on the uptake of muddy cards.