Dataset Open Access

Dataset for Publication "Learning to Exploit Multiple Vision Modalities by Using Grafted Networks"

Hu, Yuhuang; Delbruck, Tobi; Liu, Shih-Chii

Novel vision sensors such as thermal, hyperspectral, polarization, and event cameras provide information that is not available from conventional intensity cameras. An obstacle to using these sensors with current powerful deep neural networks is the lack of large labeled training datasets. This paper proposes a Network Grafting Algorithm (NGA), where a new front end network driven by unconventional visual inputs replaces the front end network of a pretrained deep network that processes intensity frames. The self-supervised training uses only synchronously-recorded intensity frames and novel sensor data to maximize feature similarity between the pretrained network and the grafted network. We show that the enhanced grafted network reaches competitive average precision (AP50) scores to the pretrained network on an object detection task using thermal and event camera datasets, with no increase in inference costs. Particularly, the grafted network driven by thermal frames showed a relative improvement of 49.11% over the use of intensity frames. The grafted front end has only 5--8% of the total parameters and can be trained in a few hours on a single GPU equivalent to 5% of the time that would be needed to train the entire object detector from labeled data. NGA allows new vision sensors to capitalize on previously pretrained powerful deep models, saving on training cost and widening a range of applications for novel sensors.

 

This page consists of selected trained models and data for evaluation.

This work was funded by the Swiss National Competence Center in Robotics (NCCR Robotics).
Files (2.4 GB)
Name Size
ECCV_NGA_data_public.zip
md5:059cb35bd7461c41a17295c81dcd56a1
2.1 GB Download
ECCV_NGA_Trained_models_public.zip
md5:41db8f023c2f2a733f10299c3f7327f4
309.4 MB Download
33
1
views
downloads
Views 33
Downloads 1
Data volume 2.1 GB
Unique views 24
Unique downloads 1

Share

Cite as