Causal Interpretation of Neural Networks

Abstract

In recent years, neural networks (NNs) have achieved great success in various challenging tasks. However, due to their black-box nature, it is still challenging to interpret how they process the data. As a result, NNs misled by spurious correlations often fail on out-of- distribution (o.o.d.) samples. Causal theory, on the other hand, has shown great potential in battling spurious correlations and interpreting complex systems. In this work, we use causal theory to tackle two tasks: NN modularity and causal representation learning. First, inspired by structural causal models (SCM), we propose a structural mechanism identification framework, namely, competitive disentanglement, to discover modular structures in NNs. The idea is to let neurons compete with each other in a specific setting such that neurons belonging to different modules will exhibit different behaviors. By observing these behaviors of neurons, we are able to disentangle them into different modules. Second, we explore the formulation of causal representation learning. The probability of causation (POC) has been regarded as a good measure of causal relevance. However, due to its counterfactual form, people have not been able to estimate the POC of representations consistently. We propose to model the representation learning process with a disentangled Variational Autoencoder (VAE) capable of producing counterfactual data that are rare in the training dataset. Thanks to the generative model in VAE, we are able to estimate the counterfactual POC directly and use them as part of the optimization goal to learn features with good causal properties.