Towards Geometry-Grounded World Understanding

Abstract

Understanding the three-dimensional (3D) world underpins embodied artificial intelligence, enabling robotics, autonomous driving, augmented and virtual reality, and, more broadly, spatial intelligence.

Yet the most direct representation of a 3D scene can be deceptively simple: a point cloud, \ie, a set of Cartesian coordinates sampled from scene surfaces, from which objects and semantic structure must be inferred.

In practice, point clouds are noisy, incomplete, and irregularly sampled, making reliable interpretation fundamentally challenging.

Geometry-grounded world understanding demands semantics that are consistent with metric 3D geometry.

Semantic segmentation provides a principled route from geometric measurements to high-level scene understanding.

In 3D point clouds, semantic segmentation anchors geometry-grounded world understanding by coupling fine-grained semantics with explicit 3D geometry and confronting a core perceptual challenge: forming structured and coherent interpretation from noisy, incomplete, and irregular samples.

This thesis investigates how explicit 3D geometry can be elevated from a passive input to a source of structure for learning and generalization.

We treat geometry as a prior that constrains what a plausible segmentation should look like, modulates how noisy supervision should be used, and guides how models adapt when spatial statistics shift.

Building on this view, we advance geometry-grounded scene segmentation along three complementary aspects of the learning problem: the output, the supervision, and the adaptation

Overall, this thesis approaches scene segmentation by structuring the outcomes of learning, the supervision that guides learning, and the contexts in which learning occurs, all grounded in explicit 3D geometry.

In doing so, we advance a geometry-grounded view of world understanding in which explicit 3D geometry shapes both learning and generalization.