Adapting Neural Architecture Search for Efficient Deep Learning Models

Abstract

This thesis presents a comprehensive investigation into Neural Architecture Search (NAS), an instrumental strategy in the formulation of proficient deep learning models. The study scrutinizes two distinct paradigms of architecture search: channel number search and operation search. In the context of channel number search, we introduce the bilaterally coupled supernet, dubbed BCNet, alongside CafeNet, endowed with a flexible weight-sharing strategy. Regarding operation search, the research puts forward K-shot NAS, featuring a K-shot supernet and reparameterization strategies. Additionally, with the objective of eliciting optimal solutions from an expansive search space, we propose the integration of Monte-Carlo Tree Search, an approach conceived to augment search efficiency and performance. Moreover, this research devises a cyclical weight-sharing strategy explicitly for the Vision Transformer architecture, and further customizes the transformer supernet training strategy. By delving into a plethora of architectures and methodologies, this thesis aspires to lay a robust foundation for future research endeavors in this domain.