Self-Supervised Intrinsic Representation Learning on Medical Images for Universal Foundation Models

Abstract

Deep learning has advanced medical imaging, yet clinical adoption is limited by scarce annotations, weak interpretability, and poor generalization. A key limitation of existing methods is their purely data-driven nature, which overlooks intrinsic properties of medical images such as anatomical symmetry, spatial hierarchies, and disease-specific structural priors. This thesis proposes a unified framework that embeds these priors into model architectures and training objectives, improving data efficiency, robustness, and interpretability.

First, I introduce a Symmetry-Aware Cross-Attention (SACA) module for brain disease diagnosis with limited supervision. By performing cross-attention between original and flipped 3D brain volumes and applying contrastive pretraining, SACA captures hemispheric asymmetries and yields improved classification and lesion segmentation on multi-center MRI datasets.

Second, I develop MSFormer, a multi-scale vision–language transformer for medical visual question answering. With Multi-Scale Positional Embedding and Grouped Attention, MSFormer fuses coarse anatomical context with fine-grained lesion details and aligns visual evidence with clinical questions, achieving state-of-the-art performance on medical VQA benchmarks.

Finally, I propose MSCAMA, a scale-aware multi-agent framework that unifies hierarchical visual encoding, adaptive retrieval, and evidence-grounded reasoning using a scale-aware backbone and specialized agents for report understanding, pairwise comparison, and question answering. Experiments demonstrate consistent improvements in retrieval accuracy, clinical relevance, and reasoning quality.

Together, these contributions advance interpretable, generalizable, and data-efficient medical imaging AI.