Rethinking Object Detection Framework through the Lens of Proposal Refinement

Abstract

With the transformative impact of the Transformer architecture, DETR pioneered the application of the encoder-decoder framework to object detection. Subsequent research, such as Deformable DETR, has aimed to enhance DETR while maintaining the encoder-decoder design. In this thesis, we revisit the DETR series through the lens of Faster R-CNN. We discover that DETR aligns with the underlying principles of Faster R-CNN's RPN-refiner design but gains advantages in end-to-end detection through the incorporation of Hungarian matching.

We systematically adapt Faster R-CNN towards Deformable DETR by integrating or repurposing each component of Deformable DETR within the Faster R-CNN framework. Our thorough analysis demonstrates that Deformable DETR's improved performance over Faster R-CNN is primarily attributable to the adoption of advanced modules, such as a superior proposal refiner that utilizes deformable attention mechanisms instead of traditional techniques like RoI Align. By viewing DETR through the RPN-refiner paradigm, we explore various proposal refinement techniques, including deformable attention, cross attention, and dynamic convolution. Each of these proposal refiners offers unique strengths in accurately refining object proposals by dynamically adjusting the focus and processing regions of interest. Our empirical studies indicate that these proposal refiners complement each other effectively, leading us to synergistically combine them into a Hybrid Proposal Refiner (HPR), which leverages the strengths of each refinement technique to enhance overall detection performance.

Our HPR is designed to be highly versatile and can be seamlessly incorporated into various DETR-based detectors, enhancing their detection capabilities. For instance, by integrating HPR into a strong DETR detector, we achieve an impressive Average Precision (AP) of 54.9 on the COCO benchmark, utilizing a ResNet-50 backbone and a 36-epoch training schedule.