Published June 18, 2026
| Version v1
Preprint
Open
Coverage Audits for LLM Interpretability Claims
Description
A Mercury methods note arguing that sparse probing, activation patching, subspace intervention, and pruning claims should report a coverage audit before making broad localization claims. The audit uses 19 fullgrid manifests and shows a 1/7 pass rate for older sparse or coarse protocols versus 12/12 for per-channel quantile dense runs.
Notes
Author ORCID: https://orcid.org/0009-0006-6816-9891. Draft prepared from local Mercury fullgrid manifests under observations/2026-05-29-fullgrid-sync/.
Files
mercury-coverage-audits-for-interpretability-zenodo-ready.zip
Additional details
Related works
- Is supplemented by
- Software: https://github.com/norika1207-lab/namecard-app (URL)
References
- Gurnee, Wes; Nanda, Neel; Pauly, Matthew; Harvey, Katherine; Troitskii, Dmitrii; and Bertsimas, Dimitris. Finding Neurons in a Haystack: Case Studies with Sparse Probing. arXiv preprint arXiv:2305.01610, 2023. https://arxiv.org/abs/2305.01610
- Zhang, Fred; and Nanda, Neel. Towards Best Practices of Activation Patching in Language Models: Metrics and Methods. ICLR, 2024. https://arxiv.org/abs/2309.16042
- Makelov, Aleksandar; Lange, Georg; Geiger, Atticus; and Nanda, Neel. Is This the Subspace You Are Looking for? An Interpretability Illusion for Subspace Activation Patching. ICLR, 2024. https://arxiv.org/abs/2311.17030
- Men, Xin; Xu, Mingyu; Zhang, Qingyu; Wang, Bingning; Lin, Hongyu; Lu, Yaojie; Han, Xianpei; and Chen, Weipeng. ShortGPT: Layers in Large Language Models are More Redundant Than You Expect. Findings of ACL, 2025. https://arxiv.org/abs/2403.03853
- Ashkboos, Saleh; Croci, Maximilian; Nascimento, Marcelo Gennari do; Hoefler, Torsten; and Hensman, James. SliceGPT: Compress Large Language Models by Deleting Rows and Columns. ICLR, 2024. https://arxiv.org/abs/2401.15024
- Carter, Shan; Armstrong, Zan; Schubert, Ludwig; Johnson, Ian; and Olah, Chris. Activation Atlas. Distill, 2019. https://distill.pub/2019/activation-atlas/
- Nanda, Neel; and Bloom, Joseph. TransformerLens. Software repository, 2022. https://github.com/TransformerLensOrg/TransformerLens