Neural Architecture Search for Convolutional and Transformer Deep Neural Networks

Abstract

Deep Neural Networks (DNN) have dominated computer vision tasks during the last decade. However, in recent years, the demand for highly customized deep neural networks has increased significantly, which has made manual design of architectures a tedious and time-consuming process. Neural architecture search (NAS), which aims to find the optimal network architecture automatically, has significantly improved network performance in many computer vision tasks. Convolutional Neural Networks (CNN)-based architecture has been the mainstream architecture in computer vision tasks. Although many NAS methods have improved the performance of CNNs, the computation cost of these methods is not affordable by most researchers, which is about thousands of GPU hours. To this end, we propose a new indicator in the CNN neural architecture search process that significantly reduces the time required by model training and evaluation during the NAS in Chapter 2. Recently, transformers without CNN-based backbones have been found to achieve impressive performance for image recognition. We propose the first neural architecture search method for the Vision Transformer models and improves the performance of the ViT models in Chapter 3. We introduce convolutional layers and propose a new hierarchical NAS method to tackle the huge search space. Finally, in Chapter 4, we introduce a new search space for ViT model and utilize the NAS method to find the optimal architecture. We find the redundancy among transformer blocks and propose a new Attention-Sharing technique to improve the network capability. We also utilize NAS techniques to decide whether each block should have the sharing attention. To sum up, we propose methods for deep neural network architecture search, including CNN and ViT models. We intend to enhance the effectiveness of NAS methods and the performance of existing models. We hope the works will contribute to the field and make NAS methods affordable to all researchers.