Multimodal Medical Image Analysis for Cancer Diagnosis and Prognosis

Abstract

Multimodal medical imaging combines techniques such as computed tomography (CT), magnetic resonance imaging (MRI), and positron emission tomography (PET) to enhance cancer diagnosis and prognosis by integrating complementary information: anatomical detail from CT, soft-tissue contrast from MRI, and metabolic activity from PET. Multiparametric MRI (mpMRI) extends this concept by combining multiple MRI sequences. While these approaches improve tumor detection, characterization, staging, and treatment monitoring, the complexity and volume of data hinder interpretation. Deep learning (DL)-based decision support systems (DSS) can integrate multimodal data to detect subtle features and complex patterns. However, volumetric data anisotropy, scarcity of annotated datasets, and the challenge of integrating diverse clinical knowledge remain barriers to robust, interpretable solutions. This thesis proposes a novel DL framework for volumetric multimodal imaging in cancer diagnosis and prognosis, featuring: (1) an anisotropic convolutional neural network (CNN) to balance dense in-plane and sparse inter-plane learning; (2) a self-supervised learning (SSL) pre-training scheme using an image restoration task; (3) a directional gradient smoothing filter to enhance SSL for anisotropic images; and (4) incorporation of prior clinical knowledge and indicators to boost performance. The framework was evaluated on three datasets: prostate cancer bi-parametric MRI (bpMRI) for clinically significant prostate cancer detection; liver lesions mpMRI for seven-class focal liver lesion diagnosis; and a private pancreatic ductal adenocarcinoma (PDAC) 18F-FDG PET/CT dataset for tumor segmentation and early recurrence prediction. Results demonstrated consistent outperformance over state-of-the-art methods across all tasks.