A Comprehensive Exploration of Video Understanding: Perspectives on Sampling, Backbone, Representation, and Cross-Modal Learning

Abstract

Video understanding is a fundamental area in computer vision with applications in autonomous driving, security, healthcare, and entertainment. As video becomes a dominant medium for information exchange, automatic interpretation is increasingly essential. While action recognition has been the foundation of video understanding, recent multimodal advancements have expanded the field to tasks like video-text matching and video question answering.

This thesis explores key advancements across multiple dimensions. First, it optimizes video recognition through salient frame selection, introducing Non-saliency Suppression Network (NSNet) to enhance efficiency and accuracy. It then investigates video model backbones, proposing the Arithmetic Temporal Module (ATM)—a plug-and-play component for temporal modeling compatible with CNNs and vision transformers.

In self-supervised video representation learning, the Macro-to-Micro Semantic Correspondence (MaMiCo) task improves learning in the absence of labeled data. Moving toward weakly supervised learning, the Text4Vis framework adapts vision-language models for video recognition by leveraging text embeddings as classifiers, enhancing zero-/few-shot recognition.

To incorporate real-world textual metadata, Cap4Video demonstrates that auxiliary captions improve text-video retrieval and recognition. Lastly, addressing the gap in video-based multimodal large language models (MLLMs), this thesis introduces Dense Connector, a plug-and-play module that enhances vision-language integration, and FreeVA, a training-free extension of image-based MLLMs to video, achieving state-of-the-art performance.

These contributions collectively advance video understanding, providing insights and practical approaches for future research and applications.