Light Field Image Reconstruction and Super-resolution

Abstract

Light field images provide richer visual information with multiple light rays to capture the appearance of objects from different directions. However, light field cameras have suffered from the inherent trade-off between angular and spatial resolution. Two tasks are focusing on alleviating this limitation, namely light field reconstruction and light field super-resolution. In this research, we study these problems and propose novel methods for the two light field tasks. The thesis falls into four parts: Firstly, to obtain deep features but avoid solely stacking convolution layers to build a very deep network, we propose U-SAS-Net for light field reconstruction which combines the merits of U-Net and separable angular and spatial convolutions (SAS). Secondly, we take advantage of the attention mechanism to propose two attention modules, namely channel-wise and SAI-wise attention modules, and embed them into U-SAS-Net to form a Channel-wise and SAI-wise Attention Network (CSANet). The attention modules can help the baseline obtain visible performance improvement at small extra costs of memory and computation. Thirdly, we study the domain asymmetry existing in the light field images and refine SAS with extra spatial convolutions. To further improve the spatio-angular feature representation, we adopt dense connections for both spatial and angular domains and propose the novel Spatio-Angular Dense Network (SADenseNet) for light field reconstruction. Finally, we investigate the previous decomposition methods' deficiency and propose unified decomposition kernels with intra-domain and inter-domain connections embedded in the novel Decomposition Kernel Network (DKNet) for light field super-resolution. Furthermore, we generalize the feature loss, which is originally for single images, to guide the network to generate visually pleasing results with more textures.