Intern-GS: Vision Model Guided Sparse-View 3D Gaussian Splatting

Abstract

Sparse-view scene reconstruction often faces significant challenges due to the constraints imposed by limited observational data. These limitations result in incomplete information, leading to suboptimal reconstructions using existing methodologies. To address this, we present Intern-GS, a novel approach that effectively leverages rich prior knowledge from vision models to enhance the process of sparse-view Gaussian splatting, thereby enabling high-quality scene reconstruction. Specifically, Intern-GS utilizes vision foundation models to guide both the initialization and the optimization process of 3D Gaussian splatting, effectively addressing the limitations of sparse inputs. In the initialization process, our method employs DUSt3R to generate a dense Gaussian point cloud. This approach significantly alleviates the limitations encountered by traditional structure-from-motion (SfM) methods, which often struggle under sparse-view constraints. During the optimization process, vision foundation models predict depth and appearance for unobserved views, refining the 3D Gaussians to compensate for missing information in unseen regions. We have tested Intern-GS across a wide range of datasets, encompassing both forward-facing and large-scale scenes. Our experiments demonstrate that Intern-GS consistently achieves state-of-the-art rendering quality on the LLFF dataset, the DTU dataset, and the Tanks and Temples dataset.