Model-Adaptive Component Designs for Efficient Deep Neural Network Training

Abstract

Although some studies have been devoted to improving model training efficiency, they rarely perform adaptive customization of such mechanisms in conjunction with model. As is well known, different components and modalities often require distinct learning rate schedules or optimization algorithms to resolve compatibility issues during training. This naturally raises an important question: should training efficiency strategies also be adapted to the interactions of model components? We take a step further by exploring component-collaborative training efficiency strategies.

Our study begins by investigating the integration of adaptive components within vision-language models (VLMs), aiming to enhance training efficiency by explicitly exploiting the unique cross-modal interactions.

First, we introduce a re-weighting mechanism. The use of dual teachers ensures that each sample receives appropriate learning guidance. Moreover, we propose a novel graph-based multimodal proximity distillation approach.

Second, we design a multimodal Mixture-of-Experts that jointly exploits image and text inputs. For each task, only a subset of experts needs to be updated, significantly reducing training time and GPU burden and thereby improving overall efficiency.

For unimodal models, our study focuses on lightweight, input/output-level adaptive components that are decoupled from the backbone.

Third, we investigate the tokenizer. We propose a simple yet effective tokenizer learning method tailored to each LLM. Customizing a tokenizer for each model incurs much lower computational costs, significantly reducing training costs.

Last, we explore visual prompts as a lightweight and efficient adaptation strategy that only modifies the input layer. We propose an Optimal Transport-based Label Mapping strategy to minimize distribution shifts and reduce the adaptation burden on visual prompts.