Automated Conversational Analysis System

Abstract

This thesis presents a comprehensive exploration into the integration of automatic speech recognition (ASR) systems with multimodal conversational analysis, structured into three distinct parts. In the first part, the thesis reviews existing methodologies for visualizing dyadic conversations and evaluates various ASR services. The development of the MONAH system incorporates a broad range of nonverbal cues into conversational analysis, which is underscored by the recent evolution of enabling artificial intelligence technologies, such as computer vision and speech-to-text. The system enhances the ability for humans to understand conversations better, potentially beneficial for populations like the hearing or sight impaired or those who need to analyze conversations at scale, for example call centers.

The advancements in the system, from its initial version to the final state-of-the-art MONAHv3, demonstrate significant improvements over traditional manual transcription methods in terms of accuracy, usability, and cost-effectiveness. In the second part, these systems have been rigorously tested in human studies against existing manual methods. The reduction in cost and turnaround time compared to manual methods, such as the expert Jefferson transcription, highlights MONAH's potential to make multimodal transcription accessible for applications, including real-time accessibility services and enhanced analysis in customer service environments.

In the third part of the thesis, the exploration of multi-task learning techniques as a peripheral research component extends the capabilities of emotion recognition systems using preprocessed multimodal data. This area, while exploratory, further enriches the functionality of multimodal annotations and sets a structured framework for future developments in automated systems. Given the rapid development of AI capabilities between 2017 and 2024, guidelines that offer novel insights for advancing automatic multimodal systems.