CLIP-Based Model Alignment Degradation on MSCOCO Retrieval Under Frequency-Domain Adversarial Noise

Benchmark accuracy is often implicitly assumed to reflect grounded visual understanding in vision-language models (VLMs), yet it remains unclear to what extent such scores truly reflect reliance on visual evidence. Motivated by a surprising observation that removing a substantial fraction of image tokens only degrades model performance very slightly on a widely used hallucination benchmark, we systematically investigate this mismatch in a set of open-source VLMs. Our analysis spans multiple levels of granularity, spanning global visual degradation, localized occlusion, question reformulation,

Research goal: To what extent does adversarial noise in the frequency domain degrade the alignment performance of CLIP-based models on the MSCOCO retrieval benchmark?

Autonomous synthesis report generated by SOVEREIGN Research Kernel. Tribunal consensus score: 8.3/10.