Robust Representation Learning: Understanding the Role of Early Stopping amidst Noisy Labels

Abstract

As large models grow in popularity, the demand for data has skyrocketed. This surge in demand has surpassed the availability of high-quality annotated data and even outstripped the volume humans can realistically produce. Consequently, training now often relies on vast amounts of data sourced directly from the Internet, which inevitably includes a significant portion of inaccurately or incorrectly labeled data. It has been observed that deep neural networks have a tendency to overfit to such noisy data, resulting in compromised generalization ability. To mitigate this challenge, a variety of robust strategies have been explored and put forward. Among these, methods based on early stopping have stood out by consistently achieving substantial success.

Despite the pivotal role of early stopping in addressing label noise, a comprehensive understanding of its intricacies remains elusive. This thesis endeavors to bridge this gap, offering several novel contributions that deepen our insight into early stopping. First, it introduces an algorithm specifically designed to harness the strengths of early stopping, enabling the extraction of hard confident examples from noisy datasets. Next, it reveals that the adverse effects of mislabeled examples become more pronounced in the layers nearing the output and proposes a refined early stopping approach via a progressive stopping strategy. Additionally, this thesis broadens the applicability of early stopping, applying it to self-supervised learning with the goal of overcoming semantic shift challenges. Finally, it provides an empirical examination the constraints of early stopping, seeking solutions for the issues posed by real-world label noise.