Event-based 3D Reconstruction: Innovative Event Representation Methods

Abstract

The event camera is a bio-inspired asynchronous brightness-change sensor capable of performing various computer vision tasks such as object tracking, object recognition, depth estimation, etc. The sparse event stream generated by the event camera contains pixel coordinates, timestamps, and polarity information corresponding to changes in brightness. Event representation refers to the specialised data preprocessing and arrangement on the event stream, which facilitates feature information extraction. Many types of event representations have been developed, but each has characteristics that vary in their applicability to different tasks. Meanwhile, many studies have attempted to use event cameras for 3D reconstruction tasks. Stereo event cameras have been more widely used in 3D reconstruction, mainly for real-time semi-dense reconstruction by calculating disparity information. On the other hand, methods based on monocular event cameras are fewer and can be divided into geometry-based real-time semi-dense reconstruction methods and deep learning-based dense reconstruction methods. Most of these methods require physical prior information, such as trajectories, and rely on a pipeline for arranging the procedures. Recently, the E2V method innovatively proposed a direct deep learning-based approach for dense voxel 3D reconstruction without requiring physical priors or a pipeline, which is a complete simplification of the event processing structure. We use the E2V method as the experimental baseline and aim to improve it through event representation. In this study, we propose five new event representation methods: Differencing Event Frame, Sobel Event Frame, Differencing Sobel Event Frame, High-Pass Event Frame, and Differencing High-Pass Event Frame. These methods enhance edge information in event frames to improve the reconstruction accuracy of the E2V method. Experiments show that our proposed best event representation method can improve the mIoU by 54% and F-Score by 38% compared to the E2V method under the same experimental conditions, which is a significant improvement. Additionally, we are the first to propose that the binarisation threshold in deep learning-based event-driven 3D reconstruction tasks should be dynamically selected for optimal performance.