Maneuvering the Sampling Process of Generative Models for Feature Alignment and Accelerated Inference

Abstract

Generative Models (GMs) have become increasingly prominent in both applied and fundamental research, with applications spanning nearly every aspect of daily life. Despite their remarkable success in producing high-quality and realistic outputs, GMs still face fundamental challenges during the sampling process. In many cases, even when using the same model, the generated samples can vary dramatically—some closely align with predefined conditions and exhibit excellent quality, while others deviate significantly and appear poor or unrecognizable.

This thesis addresses the central question: How can we control the sampling process to avoid poor outputs and consistently generate high-quality features? We approach this problem through an optimization-based framework. First, we formulate the sampling process as an optimization problem. Then, depending on the specific task, we design different objective functions to systematically improve various aspects of the sampling process across different types of generative models. The proposed methods are benchmarked on multiple baselines, covering both diffusion-based and autoregressive generative models.

Our results demonstrate that the optimization-based perspective provides a unified and effective approach for enhancing the quality and consistency of samples across diverse generative modeling paradigms.