Enhancing Medical Vision-Language Foundation Models in Tumour Malignancy Recognition via Pre-trained Language Models

Abstract

Vision-Language Foundation Models (VLFMs) offer strong potential for zero-shot learning in histopathology, where models must recognise rare or unseen disease subtypes with limited annotation. However, pathology VLFMs often rely on coarse textual descriptions that miss fine-grained visual cues, causing weak image-text alignment, noisy retrieval, and reduced classification accuracy. This thesis proposes Retrieval-based De-noising Causal Language Modelling (RDCLM), a framework for improving zero-shot tumour malignancy recognition by refining noisy retrieval outputs from pathology VLFMs. RDCLM builds a pathology-specific knowledge base of discriminative benign and malignant tumour descriptions using a large language model. For each query histopathology image, a pathology VLFM retrieves candidate descriptions from this knowledge base, and a frozen pre-trained language model integrates the retrieved text with projected visual features to suppress irrelevant content and retain malignancy-relevant evidence. To improve robustness, two retrieval augmentation strategies are introduced: Retrieval Negatives Replacement (RNR) and Description-wise Shuffling (DS). A Multi-Branch Diverse-Dimension Projection architecture with an auxiliary Inter- and Intra-Branch Min-Max Mutual Information Optimisation objective is also proposed to encourage diverse and informative cross-modal feature learning. Precision Reward Loss is further evaluated as an auxiliary refinement for cleaner de-noised descriptions. Experiments on five histopathology cancer datasets show that RDCLM improves both zero-shot image-text retrieval and image classification compared with state-of-the-art CLIP-based, retrieval-based, and unimodal baselines. The results demonstrate that retrieval de-noising with pre-trained language models can strengthen semantic alignment and improve VLFM-based malignancy recognition in histopathology.