Probabilistic Machine Learning in the Age of Deep Learning: New Perspectives for Gaussian Processes, Bayesian Optimisation and Beyond

Abstract

Advances in artificial intelligence (AI) are rapidly transforming our world, with systems now surpassing human capabilities in numerous domains. Much of this progress traces back to machine learning (ML), particularly deep learning, and its ability to uncover meaningful patterns in data. However, true intelligence in AI demands more than raw predictive power; it requires a principled approach to making decisions under uncertainty. Probabilistic ML offers a framework for reasoning about the unknown in ML models through probability theory and Bayesian inference. Gaussian processes (GPs) are a quintessential probabilistic model known for their flexibility, data efficiency, and well-calibrated uncertainty estimates. GPs are integral to sequential decision-making algorithms like Bayesian optimisation (BO), which optimises expensive black-box objective functions. Despite efforts to improve GP scalability, performance gaps persist compared to neural networks (NNs) due to their lack of representation learning capabilities. This thesis aims to integrate deep learning with probabilistic methods and lend probabilistic perspectives to deep learning. Key contributions include: (1) Extending orthogonally-decoupled sparse GP approximations to incorporate nonlinear NN activations as inter-domain features, bringing predictive performance closer to NNs. (2) Framing cycle-consistent adversarial networks (CYCLEGANs) for unpaired image-to-image translation as variational inference (VI) in an implicit latent variable model, providing a Bayesian perspective on these deep generative models. (3) Introducing a model-agnostic reformulation of BO based on binary classification, enabling the integration of powerful modelling approaches like deep learning for complex optimisation tasks. By enriching the interplay between deep learning and probabilistic ML, this thesis advances the foundations of AI, facilitating the development of more capable and dependable automated decision-making systems.