Journal article Open Access

Multimedia retrieval based on non-linear graph-based fusion and partial least squares regression

Gialampoukidis, Ilias; Moumtzidou, Anastasia; Liparas, Dimitris; Tsikrika, Theodora; Vrochidis, Stefanos; Kompatsiaris, Ioannis

Heterogeneous sources of information, such as images, videos, text and metadata are often used to describe different or complementary views of the same multimedia object, especially in the online news domain and in large annotated image collections. The retrieval of multimedia objects, given a multimodal query, requires the combination of several sources of information in an efficient and scalable way. Towards this direction, we provide a novel unsupervised framework for multimodal fusion of visual and textual similarities, which are based on visual features, visual concepts and textual metadata, integrating non-linear graph-based fusion and Partial Least Squares Regression. The fusion strategy is based on the construction of a multimodal contextual similarity matrix and the non-linear combination of relevance scores from query-based similarity vectors. Our framework can employ more than two modalities and high-level information, without increase in memory complexity, when compared to state-of-the-art baseline methods. The experimental comparison is done in three public multimedia collections in the multimedia retrieval task. The results have shown that the proposed method outperforms the baseline methods, in terms of Mean Average Precision and Precision@20.

Files (1.3 MB)
Name Size
MTAP_submit_zenodo.pdf
md5:ae0ac4b6be64d36ce278360174738fea
1.3 MB Download
90
78
views
downloads
Views 90
Downloads 78
Data volume 100.8 MB
Unique views 87
Unique downloads 77

Share

Cite as