3D Human Motion Recovery and Generation: From Noisy Pose to Language Description

Abstract

Understanding and generating human behaviors are crucial and fundamental to numerous computer vision applications, including embodied AI, autonomous driving, and human-machine interaction. In the field of human behavior modeling, there are two primary components: perception and generation. Perception involves detecting 2D and 3D poses from sensor data to interpret and understand human behaviors in real-world scenarios. In contrast, generation focuses on creating realistic or desired human behaviors based on conditional signals, such as natural language directions, offering significant advantages in animation creation, filmmaking, and robot control. Within human behavior modeling, this thesis focuses on three critical aspects of perception and generation: robust pose estimation from images, realistic motion recovery from monocular videos and noisy inputs, and interactive motion synthesis conditioned on natural language descriptions. We address these three key areas in human behavior research, proposing solutions that lay the foundation for a comprehensive human motion framework in future work. It will pave the way for applications in VR/AR, animation, healthcare, and human-robot interaction.