Deep Learning-based Methods for Sequential 3D Point Cloud Upsampling

Abstract

Point cloud is a fundamental 3D representation that has received increasing attention from the computer vision community due to the recent popularity of Light Detection and Ranging (LIDAR) sensors and RGB-D cameras. However, analyzing and understanding 3D point clouds with deep networks is challenging since the raw point cloud data captured by scanning devices always suffers from sparsity and irregular data structure. To cope with this problem, several studies have been proposed to perform restoration by upsampling the input sparse point clouds via deep neural networks (DNN). Despite the successes achieved by these studies, they cannot generate satisfying results due to the insufficient geometry contexts from the input single point cloud snapshot. Considering that the 3D sensors usually capture point cloud sequences instead of each single point cloud snapshot, in this thesis, I propose three sequential point cloud sampling algorithms to significantly improve the performance of the previous single frame-based methods by exploiting temporal dependencies. In the first method, I propose a recurrent framework for sequential point cloud upsampling (SPU), which leverages multi-scale long-/short-term temporal dependencies to recover the point cloud of the current frame. To improve the efficiency and generalizability, in the second method, I extend the sequential point cloud upsampling framework to a patch-based version, namely the VPU framework. Lastly, to address the patch-misalignment issue from VPU, I propose a sequential point cloud upsampling framework built upon progressive multi-level refinement (PMR-Net), including three different refinement stages at the frame level, shape level, and detail level, respectively. Comprehensive experiments on multiple point cloud sequence datasets demonstrate the effectiveness of our proposed frameworks SPU, VPU, and PMRNet for sequential point cloud upsampling.