Visually Rich Document Understanding and Intelligence

Abstract

Visually Rich Documents (VRDs) are potent carriers of multimodal information widely used in academia, finance, medical fields, and marketing. Traditional approaches to extracting information from VRDs rely on expert knowledge and manual labour based on predefined rules, leading to high costs. However, significant gaps still exist between current research focuses and the real-world nature of VRDs. Most existing research focuses on simple or well-structured domains, such as receipts and academic papers, often limited to single-page scenarios. This research introduces three research directions to address these gaps: 1) Neglect of Complex VRD Structures, 2) Lack of Multi-Page Scenarios Exploration, and 3) Lack of Exploration in Large Model Integration. For complex format VRD understanding, a new dataset for complex form structure and key information extraction is introduced to address complex form structures and multi-party(authors) information gaps. A new key information extraction baseline is introduced with multi-aspect features. To address the challenges in multi-page VRD understanding, a new dataset for multimodal, multi-page VRD question answering is proposed to retrieve multimodal document semantic entities across several pages based on a question. A set of frameworks are proposed to leverage various pretrained models to enhance the representative of multi-page document representations. Finally, to address the gaps in knowledge transfer from large-scale models for VRD understanding, a new knowledge distillation-based framework is proposed to distil knowledge from multiple large-scale models by introducing multi-task losses to transfer multi-teacher knowledge to lightweight student models, achieving more comprehensive document representation.