Accurate object detection using the sensor fusion of an event-based and a frame-based camera

Efficient object detection is crucial to real-time monitoring applications such as autonomous driving.
Modern RGB cameras can produce high-resolution images for accurate object detection. However, with
increased resolution comes increased network latency and power consumption. To minimise this latency,
CNNs often have a resolution limitation, requiring images to be down-sampled before inference, causing
significant information loss. In this paper, we propose a neuromorphic vision approach based on biological
vision, where images are cropped instead of down-sampled to meet the requirements of the CNN. The design
implements the sensor fusion of an event-based camera and a frame-based RGB camera to implement an
accurate, low-power monitoring system. The cameras are calibrated to create a multi-modal stereo vision
system where pixel coordinates can be projected between the event camera and RGB camera image planes.
Events are detected using clustering on the event-based data and projected to the RGB image plane using
the calibration results. These projections identify important areas in the RGB image, directing the cropping
areas. Using this implementation, the COCO AP is increased from 21.08 to 57.38 on bicycles in the RGB
scene, with an overall increase from 37.93 to 46.89 for all classes tested.