Vision Neural Architecture Designs for Improved Robustness and Accuracy

Abstract

Deep neural networks (DNNs), such as Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs), have achieved impressive results in computer vision but remain vulnerable to input shifts from adversarial attacks or natural distribution changes. This undermines their reliability and limits real-world deployment. Current defences like adversarial training and distributionally robust optimization (DRO) aim to improve robustness but often sacrifice accuracy. Adversarial training reduces standard accuracy, while DRO struggles to balance in-distribution (ID) and out-of-distribution (OOD) performance. Overcoming this robustness-accuracy trade-off remains a key challenge in deep learning.

Firstly, we propose Neural Architecture Dilation (NAD), a framework that improves robustness by integrating searched dilation architectures into pre-trained neural backbones. Building on this, we introduce Neural Architecture Dilation for Adversarial Robustness (NADAR), which formulates the dilation process as a constrained optimization problem. Secondly, we introduce TORA-ViTs, a transformer-based architecture that explicitly disentangles robust and predictive features through lightweight adapters, while an attention-based gated fusion mechanism dynamically balances their contributions based on input conditions. To optimize this fusion, we implement a two-phase training strategy. Thirdly, we propose EdgeNet, a plug-and-play module designed to enhance robustness by incorporating shape-based edge features into vision transformers. Utilizing a "sandwich" architecture with zero convolutions, EdgeNet introduces robust structural representations while preserving the predictive power of pretrained backbones. Lastly, we introduce AdaptNAS, a Neural Architecture Search (NAS) framework that explicitly optimizes architectures for both ID accuracy and OOD robustness. AdaptNAS leverages domain adaptation principles and adversarial learning to reduce the OOD generalization gap.