Multimodality Integration for Natural Language Generation

Abstract

Multimodal Natural Language Processing integrates text processing with other data modalities like image, video, audio and speech. Recently, developing models capable of generating human-like language conditioned on multimodal data has been a growing research area. In this thesis, I focus on two Natural Language Generation (NLG) tasks: 1) Visual Storytelling (VST) which requires a model to produce a matching story given an image sequence, and 2) Video Paragraph Captioning (VPC) where given a video prompt, a coherent textual summary describing key events in the video needs to be generated. To understand these image and video modalities, many models utilise pretrained networks to convert the raw data into an embedding stream which is then fed into a sequence-to-sequence model to decode the text. However, this fails to explicitly capture relations between important aspects such as interactions between multimodal features or temporal relationships between key events. Thus, how to best represent this complex multimodal data is an ongoing challenge. Moreover, another challenge is that models tend to lack commonsense and reasoning ability due to only being exposed to limited training data. Consequently, models become incapable of producing outputs that go beyond the pattern recognised in the data.

Therefore, to address these challenges, this research explores methods for transforming the image and video input into high-level semantic commonsense-enhanced graphs to promote scene and context understanding. Furthermore, for evaluation, due to the lack of metrics designed for VST, my thesis additionally proposes a novel reference-free metric which takes into account the fact that storytelling is subjective in nature.

As the final proposed work, a multimodal commonsense knowledge graph focusing on social and event-based knowledge is introduced. It is hoped that such multimodal commonsense knowledge can be used as auxiliary information to help improve NLG tasks like VST and VPC.