A survey of multimodal event detection based on data fusion

With the emergence of the Internet of Things (IoT) and the rise of shared multimedia content on social media networks,available datasets have become increasingly heterogeneous. Several multimodal techniques for detecting events in data ofdifferent types and formats have emerged. Those techniques implement various detection algorithms and present differenttrade-offs in terms of data fusion. Unfortunately, little is known about their underlying detection mechanisms, as existingcomparisons are limited to either unimodal event detection techniques or specific types or representations for multimodaltechniques. Understanding the behavior of multimodal event detection techniques remains an acute open research problem.In this work, we present a systematic literature review of multimodal event detection techniques. We describe how varioustechniques leverage information from different modalities through data fusion. We further propose a novel taxonomy ofmultimodal event detection techniques according to their temporal orientation and the inner workings of their detectionmechanism. Finally, we analyze the datasets and metrics used in previous works as well as their reported results. Our surveyallows to uncover the properties of each approach and discuss future research directions in this field.