Multimodal NLP in Mental Healthcare

Abstract

Mental health has been a growing concern for countries, communities, and individuals. Despite considerable advances, mental healthcare systems still face significant challenges, prompting researchers to explore opportunities in deep learning and natural language processing. However, recent research trends have shifted toward incorporating various media-based modalities, including videos, images, and physiological data. This shift, while promising, introduces new limitations, particularly in terms of data accessibility and research reproducibility. This thesis addresses these challenges by leveraging the ubiquity of textual data in mental health-related settings, aiming to exhaust different text-derived complementary information at different abstraction levels to enrich textual representations beyond standard semantic contextualisation.

The main contributions are threefold, proposing three abstraction-level modalities and three different approaches to multimodal integration to improve mental health risk detection and information extraction. First, inspired by the complexity of human emotions and language, affective information from the emotion modality is integrated through multi-emotion graph pretraining for depression and suicide risk detection. The second study introduces the acoustic modality, capturing prosodic information derived from textual data and integrating it through a multi-teacher knowledge distillation framework, along with emotion and textual abstractions, for the same mental health tasks. Finally, the word-pair modality is explored, proposing a novel perspective on relational-structural abstraction from raw textual input, integrated through a triplet-grid framework, that improves word-boundary detection for the extraction of disjointed adverse drug reactions.