Multimodal Emotion Elicitation and Recognition in Virtual Reality

Abstract

Virtual Reality (VR) has been effectively used for eliciting emotions, yet most research focuses on the intensity of affective responses rather than on how interaction influences those experiences. To address this gap, this thesis advances a validated VR emotion-elicitation dataset through two extensions. First, we add a new high-arousal, high-valence scene and validate its effectiveness in a within-subject study (N=24). Second, we create interactive and non-interactive versions of each scene to examine the impact of interaction on emotional responses. We evaluate interaction using subjective ratings and physiological signals. Our evaluation study (N=84) shows that interaction not only amplifies emotions but also modulates them in context, supporting coping in negative scenes and enhancing enjoyment in positive scenes.

Multimodal Emotion Recognition (MER) increasingly depends on fine-grained, evidence-grounded annotations, yet inspection and label construction are hard to scale when cues are dynamic and misaligned across modalities. This thesis presents an LLM-assisted toolkit that supports multimodal emotion data annotation through an inspectable, event-centered workflow. The toolkit aligns heterogeneous recordings, visualizes modalities on a shared timeline, and packages synchronized keyframes and time windows as traceable event packets. It then uses modality-specific tools and prompt templates to draft structured annotations for analyst verification and editing.

Building on the dataset extensions and annotation, this thesis further investigates MER modeling approaches in VR that integrate behavioural and physiological signals from VR headsets and wearable sensors. We introduce an LLM-based Mixture-of-Experts (MoE) framework, where experts specialize in different modalities and a router assigns weights to experts for each event. The goal is to connect predictions to traceable multimodal evidence and support interpretation of affective cues in interactive VR.