File uploads: We have fixed an issue which caused file uploads to fail. We apologise for the inconvenience it may have caused.

Published October 10, 2022 | Version v1
Journal article Open

ViGAT: Bottom-up event recognition and explanation in video using factorized graph attention network

Description

In this paper a pure-attention bottom-up approach, called ViGAT, that utilizes an object detector together with a Vision Transformer (ViT) backbone network to derive object and frame features, and a head network to process these features for the task of event recognition and explanation in video, is proposed. The ViGAT head consists of graph attention network (GAT) blocks factorized along the spatial and temporal dimensions in order to capture effectively both local and long-term dependencies between objects or frames. Moreover, using the weighted in-degrees (WiDs) derived from the adjacency matrices at the various GAT blocks, we show that the proposed architecture can identify the most salient objects and frames that explain the decision of the network. A comprehensive evaluation study is performed, demonstrating that the proposed approach provides state-of-the-art results on three large, publicly available video datasets (FCVID, MiniKinetics, ActivityNet). Source code is made publicly available at: https://github.com/bmezaris/ViGAT

Notes

This work was supported by the EU Horizon 2020 programme under grant agreements 832921 (MIRROR) and 101021866 (CRiTERIA); and, by the QuaLiSID - "Quality of Life Support System for People with Intellectual Disability" project, which is co-financed by the European Union and Greek national funds through the Operational Program Competitiveness, Entrepreneurship and Innovation, under the call RESEARCH-CREATE-INNOVATE (project code: T2EDK-00306).

Files

vigat_arxiv.pdf

Files (7.3 MB)

Name Size Download all
md5:091b11fc17f31abfb7a96c8cc3fc5ccb
7.3 MB Preview Download

Additional details

Funding

CRiTERIA – Comprehensive data-driven Risk and Threat Assessment Methods for the Early and Reliable Identification, Validation and Analysis of migration-related risks 101021866
European Commission
MIRROR – Migration-Related Risks caused by misconceptions of Opportunities and Requirement 832921
European Commission