Fast and Flexible Expectation Propagation for Linear Model Extensions

Abstract

Expectation propagation (EP) is an approximate Bayesian inference method that has gained traction over the past two decades due to its accuracy, efficiency, and amenability to distributed computation. The core idea of EP is to construct the posterior approximation as a product of approximations to the posterior factors, which are iteratively refined so as to locally minimise the Kullback-Leibler (KL) divergence from this posterior approximation to the true posterior. At a high level, this factor-wise, greedy refinement strategy improves tractability by focusing on one factor at a time, rather than attempting to minimise the KL divergence between the full posterior approximation and the true posterior in a single step. Because each factor is typically simpler than the full posterior, this local optimisation approach is computationally more manageable.

Despite its promise, EP is underused in many settings where it could offer substantial benefits. One reason is that computing the required moments of the so-called tilted distributions --- intermediate distributions formed by combining a posterior factor with the current posterior approximation --- can be analytically intractable for models with complex structures. In such cases, the standard implementation of EP becomes computationally expensive or inaccurate, limiting its practical appeal. More fundamentally, EP has mostly been viewed as a method for posterior approximation, and its potential for other inferential tasks remains underexplored.

This thesis addresses these gaps for the Bayesian versions of common extensions of linear models.