Aircraft Marshaling Signals Dataset of FMCW Radar and Event-Based Camera for Sensor Fusion

Dataset Introduction

The advent of neural networks capable of learning salient features from variance in the radar data has expanded the breadth of radar applications, often as an alternative sensor or a complementary modality to camera vision. Gesture recognition for command control is the most commonly explored application. Nevertheless, more suitable benchmarking datasets are needed to assess and compare the merits of the different proposed solutions. Furthermore, most current publicly available radar datasets used in gesture recognition provide little diversity, do not provide access to raw ADC data, and are not significantly challenging. To address these shortcomings,  we created and made available a new dataset that combines two synchronized modalities: radar and dynamic vision camera of 10 aircraft marshalling signals at several distances and angles, recorded from 13 people. Moreover, we propose a sparse encoding of the time domain (ADC) signals that achieve a dramatic data rate reduction (>76%) while retaining the efficacy of the downstream FFT processing (<2% accuracy loss on recognition tasks). Finally, we demonstrate early sensor fusion results based on compressed radar data encoding in range-Doppler maps with dynamic vision data. This approach achieves higher accuracy than either modality alone.

Dataset Structure

The dataset has a common directory structure which contains additional information about the captures.

dataset_dir/<stage>/<room>/<person>-<gesture>-<distance>/ofxRadar8Ghz_yyyy-mm-dd_HH-MM-SS.rad

Identifiers

• stage [train, test].
• room: [conference_room, foyer, open_space].
• person: [0-9]. Note that 0 stands for no person, and 1 for an unlabeled, random person (only present in test).
• gesture: ['none', 'emergency_stop', 'move_ahead', 'move_back_v1', 'move_back_v2', 'slow_down' 'start_engines', 'stop_engines', 'straight_ahead', 'turn_left', 'turn_right'].
• distance: ['xxx', '100', '150', '200', '250', '300', '350', '400', '450'] (in cm). Note that xxx is used for none gestures when there is no person present in front of the radar (i.e. background samples), or when a person is walking infront of the radar with varying distances but performing no gesture.