Learning Theory for Transformers: An Operator-Learning Viewpoint

Abstract

Large language models (LLMs) have reshaped the foundations of artificial intelligence research and the modes of interaction between human cognition and machine intelligence. Their influence extends further still, transforming the scientific tools through which we interrogate and model the physical world. Underlying most of these achievements and breakthroughs is a dominant architecture: the Transformer. Although the Transformer was proposed nearly a decade ago, established mathematical frameworks remain insufficient to explain the complex phenomena observed in practice with Transformer-based networks, particularly large language models. This thesis offers a principled theoretical foundation for understanding the remarkable capabilities these models exhibit, grounded in a central argument that the Transformer performs operator learning during pretraining over vast text corpora. Our analysis reveals the nature of pretraining and in-context learning mechanisms of efficient Transformer structures in an operator learning framework. Transformers maps each context distribution to a response function for queries and with more samples from context distribution, they can recover information as much as possible to get a better response function with fixed and pretrained weights without any update.