CroQS Benchmark

CroQS (Cross-modal Query Suggestion) v1.0.0 is a benchmark dataset designed to evaluate methods that generate improved textual queries guided by visual results, in the context of text-to-image retrieval. The dataset supports the task of generating query suggestions grounded in visual content, specifically helping users refine or reformulate queries based on result set clusters.

This version includes:

• 50 initial textual queries used to retrieve image sets from the MS COCO 2017 dataset.

• 295 manually defined semantic clusters, based on visual similarity or common properties in the image results.

• 8127 unique COCO images (referenced via URL, not redistributed directly).

• Each query result set is manually grouped into 2 to 10 clusters (average ~5.9 clusters per query).

• Each cluster is associated with a human-written query suggestion that describes the cluster's shared visual properties.

CroQS can be used to train or evaluate models for:

• Query refinement and expansion

• Multimodal and cross-modal retrieval

• Cluster-based query suggestion

• Vision-language understanding

For more details on the benchmark task and evaluation metrics, refer to our paper:

Maybe You Are Looking for CroQS: Cross-modal Query Suggestion for Text-to-Image Retrieval.
In Proceedings of the 46th European Conference on Information Retrieval (ECIR 2025).
Springer Link | arXiv:2412.13834

Website: https://paciosoft.com/CroQS-benchmark