Towards controllable neural audio synthesis: An exploration of sound effects creation with generative models

Abstract

Sound effects are crucial components in enhancing the auditory experience in media such as film, television, and video games. Traditionally, these effects are created using two primary methods: Foley recording, where sound is physically performed to match onscreen actions, and digital sound processing (DSP), which manipulates audio signals with algorithms to achieve desired sounds. While effective, these methods come with limitations in terms of scalability and the ability to rapidly prototype sounds.

Recent advancements in deep learning have introduced a new paradigm for sound effects generation through neural audio synthesis. This approach utilizes generative models to produce sound effects from learned audio features automatically. However, a significant drawback of this technology is the lack of fine-grained control over the sound output, as generative models typically provide fewer parameter adjustments compared to traditional DSP methods. This limitation poses challenges in achieving specific auditory outcomes necessary for creative sound design.

This thesis explores neural audio synthesis in sound effects generation. The data scarcity in the domain of sound effects has been a well-known issue, which makes it challenging for data-driven approaches towards learning and synthesizing. This thesis aims to study controllable and diverse sound effects generation under the setting of a limited audio dataset without requiring huge computation resources.