Self-Supervised Visual Representation Learning in Distributed Systems - Generalisation Analysis and Training Framework Design

Abstract

This thesis investigates distributed self-supervised learning as a paradigm for training visual representation models directly from distributed and unlabelled data. While large-scale supervised datasets have driven advances in visual artificial intelligence, their centralised collection and annotation are costly, and real-world data is inherently fragmented across edge devices, institutions, and sensors. Distributed self-supervised learning aims to leverage such data without labels or central coordination, raising key challenges including robustness under heterogeneous client distributions, feasibility on resource-limited devices, and whether distributed training can approach centralised performance.

This thesis makes four main contributions. First, it shows how decentralisation reshapes scaling laws, proving that the compute-optimal model size decreases as data becomes more distributed, explaining the effectiveness of lightweight models on edge devices. Second, it demonstrates that distributed training exhibits a generalisation gap compared to centralised training under equal compute, which can be reduced by increasing data through more clients or larger local datasets. Third, it provides a systematic theoretical analysis under heterogeneous data, showing that Masked Image Modelling is more robust than contrastive learning, with robustness improving with network connectivity, and introduces MAR loss to enhance robustness. Finally, it proposes DeNAV, a decentralised framework that removes server dependence and incorporates informed client selection, staleness-aware aggregation, and lightweight masked autoencoder pre-training for efficient communication, with theoretical guarantees and strong empirical performance.

Overall, this thesis establishes distributed self-supervised learning as a feasible and effective paradigm and shows how theoretical insights on scaling laws, generalisation, and robustness guide the design of practical distributed learning systems.