Criticality and Anomalous Diffusion Underlying Dynamical and Computational Properties of Deep Learning Networks

Abstract

Artificial neural networks, inspired by biological neural systems, harness modern computing power to facilitate learning. The corresponding scaled-up models, known as deep neural networks (DNNs), consistently achieve state-of-the-art performance across various user-defined and scientific applications. Their success is largely driven by the integration of diverse operators, such as linear and convolutional layers, alongside self-attention blocks. The vast number of model parameters enable extensive analysis of network properties, including width and depth, allowing DNNs to be treated as mathematical objects in theoretical studies. Drawing from neuroscience and statistical physics, we impose statistical assumptions on artificial neural networks and their learning processes. In particular, heavy-tailed, power-law distributions prevalent in many complex systems, give rise to heterogeneity in neural connectivity, increasing the likelihood of extreme connection strengths. Leveraging heavy-tailed non-Hermitian random matrix theory, we reveal an extended criticality-like regime between the chaotic and ordered phases that display ideal information propagation and training dynamics, further extending classical Gaussian results. Next, we generalize the diffusion processes underlying the self-attention mechanism of Transformers to the class of Lévy processes, incorporating long-range interactions between tokens on the embedding space manifold. This enhances the self-attention performance and gives rise to fractional diffusion dynamics in its continuous space-time limit. Finally, we analyze the short- and long-term behavior of stochastic gradient descent (SGD) in various settings, revealing non-equilibrium dynamics driven by anomalous diffusion that naturally arise and benefit the deep learning process. The findings presented in this thesis illustrate the importance of developing theoretical tools for understanding shared principles governing artificial and biological systems.