Multimodal Physiological Cognitive Load Measurement

Abstract

Monitoring users’ cognitive load in real-time allows the system to adjust its interface and improve the interaction experience and user performance. Physiological signals are relatively reliable, real-time measures of cognitive load. Measurement robustness can be improved by taking account of confounding factors, and multimodality has the potential to enhance mental load prediction. This thesis investigates cognitive load measurement by means of physiological data and machine learning techniques. Skin response and eye blink are economical, conveniently-captured physiological measures that were studied. Multiple datasets were used to increase the reliability of the results which confirmed that the explored features can significantly measure different cognitive load levels. Confounding factors can distort cognitive load measurement results. Emotional fluctuations have profound impacts on physiological signals. Therefore to examine the robustness of the explored features, they were evaluated for cognitive load measurement with affective interference in different datasets. The results showed that they can measure multiple cognitive load levels with high accuracy even under emotional changes. Different modalities can impart complementary information. Hence we tried to improve the accuracy by means of multimodal cognitive load measurement. Two fusion techniques were used and different combinations of classifiers and features were examined. Multimodality proved to improve the cognitive load classification accuracy and the studied feature fusions performed well both in the absence and presence of affective stimuli. This thesis proposes frameworks for monitoring cognitive load using physiological data and machine learning techniques. The system was tested during affective fluctuations and modality fusion was performed. The outcomes of this research show that the explored features and methods could be used as means for objective, robust, accurate cognitive load measurement.