On the Importance of Transition Matrix for Learning with Noisy Labels

Abstract

To improve the generalization ability of deep learning models when training data contains noisy labels, a noise transition matrix T(x) has been widely employed to reveal the transition relationship from clean labels to noisy labels of instances. It acts as a crucial building block in designing statistical-consistent methods for learning with noisy labels (T-based methods). However, for real-world datasets, the transition matrix is usually unknown and needs to be estimated. Accurately estimating the transition matrix can be a challenging task. This motivates recent work to design label-noise robust methods focusing on incorporating heuristics instead of requiring estimating the transition matrix (heuristic-based methods). The heuristic-based method has demonstrated state-of-the-art (SOTA) performance on many benchmark datasets. These methods seem to be more practical than T-based methods. It raises the question that is the transition matrix still important for learning with noisy labels.

In this thesis, we answer that the transition matrix still plays an important role in learning with noisy labels. We will show that the transition matrix not only can be used to design statistical-consistent methods but also can help boost the performance of heuristic-based methods. We will also show that given the transition matrix, the performance of T-based methods will not be influenced by different data generative processes. By contrast, the performance of SOTA heuristic-based methods can be influenced by different data generative processes. Since the label-noise transition matrix is important but hard to estimate, we will propose two new transition-matrix estimation methods that reduce the estimation error of the transition matrix. The first method can effectively estimate instance-independent transition matrix by exploiting the divide-and-conquer paradigm. The second method focuses on estimating instance-dependent transition matrices by leveraging a structural causal model.