Hidden in Top-K: Probing Vision-Language Model Understanding Through Post-Retrieval Analysis

General information

Abstract: Vision-language models (VLMs) are widely evaluated for their cross-modal understanding using handcrafted datasets with contrastive image-caption pairs. While effective for controlled comparisons, such evaluations rely on small candidate sets and limited negative examples, potentially obscuring systematic failure modes under more realistic retrieval settings. In this work, we introduce a post-retrieval analysis framework that evaluates VLMs by inspecting their top-K retrievals from large candidate pools, providing a more diagnostic view of model behavior. We evaluate five VLMs—CLIP, BLIP-2, FLAVA, SigLIP2, and finetuned Qwen2.5-VL on the SVO-Probes, targeting image–text relations such as subjects,objects and verbs. Our evaluation combines standard retrieval metrics, novel semantic-similarity metric, human judgments, and large vision–language model–based assessments to account for incomplete annotations and semantically valid alternatives. Our results indicate that while VLMs achieve high pairwise accuracy and image-text matching accuracy, they struggle in top-K retrieval, particularly with actions
and relational content (human evaluation success rate at 1 ≈ 70%). The o3 assessments closely align with human judgment, enabling scalable evaluation. Overall, our results highlight important limitations of current benchmarks and demonstrate the value of post-retrieval analysis for diagnosing robustness and semantic sensitivity in vision–language models.

Paper: currently under review process

The data are available upon request for research purposes only.

This research was supported by the EU NextGenerationEU through the Recovery and Resilience Plan for Slovakia under the project No. 09I01-03-V04-00007, DisAI-AMPLIFIED.

Tables of content

We randomly selected 100 samples from SVO-Probes. We used both the images and the texts of these samples for retrieval with all of the five mentioned VLMs. So for each model there are two json files and each json file containes entries that looks like this (from BLIP2_selected_samples_text_retrieval_evaluation_o3_experiment.json):

{
    "221": {
        "BLIP2 retrieved captions": [
            "Girl sits on a ball.",
            "A girl sits on a ball.",
            "A girl sitting on a ball.",
            "girl sits on ball",
            "A woman is sitting on a ball.",
            "the girl sits in the pool",
            "The girl can sit with no background.",
            "woman, ball, outside",
            "A girl sits on a soccer ball.",
            "One girl enjoy in a beach."
        ],
        "human evaluation": [
            "correct",
            "correct",
            "correct",
            "correct",
            "correct",
            "object incorrect",
            "correct",
            "correct",
            "object incorrect",
            "object incorrect"
        ],
        "GPT evaluation": [
            "correct",
            "correct",
            "correct",
            "correct",
            "subject incorrect",
            "object incorrect",
            "object incorrect",
            "object incorrect",
            "object incorrect",
            "object incorrect"
        ],
        "human evaluation 2": [
            "1",
            "1",
            "1",
            "1",
            "1",
            "-1",
            "1",
            "-1",
            "-1",
            "-1"
        ],
        "human evaluation 3": [
            "1",
            "1",
            "1",
            "1",
            "1",
            "-1",
            "1",
            "-1",
            "-1",
            "-1"
        ],
        "o3 evaluation": [
            "correct",
            "correct",
            "correct",
            "correct",
            "correct",
            "object incorrect",
            "correct",
            "object incorrect",
            "object incorrect",
            "object incorrect"
        ]
    }, ...