Published November 4, 2026 | Version 1.2

Semantic Conformance in Agent Report Evaluation

Authors/Creators

  • 1. Independent researcher

Description

Report-integrity evaluations depend on distinguishing what an agent reports from what independently recorded execution evidence establishes. This retrospective case study documents measurement development in a synthetic agent-workflow project. Small matched-evidence model pilots exposed classifier artifacts and motivated structured extraction of report claims. The development history covers three successor extractors; quantitative failure analysis concentrates on extractor v1.1.2. That candidate matched 343 disclosed expectations, while an AI-assisted audit in a fresh context with access to its code reported violations on 12 of 58 constructed adversarial probes. Seven failures were modal-event variants; five involved a shared segmentation weakness followed by mention-scope or quoted-negator errors. These counts describe the examined artifacts and probes, not population accuracy or a count of independently isolated causes. The report presents code-reading explanations and the decision not to adopt the extractors. Its contribution is an inspectable case record of the gap between authored calibration agreement and semantic conformance in this implementation. A separately sealed holdout remained unopened.

Report v1.0 (3 November 2026) was published on the author’s website only and was not deposited on Zenodo. Report v1.1 (4 November 2026) was the first Zenodo deposit. Report v1.2 is a separate revised edition with its own DOI, preserving the earlier record and PDF. It clarifies terminology and schema conventions, expands the qualified historical audit summary, and improves the figure, bibliography, and metadata. The underlying results are unchanged.

The deposit contains the technical report only. Historical executions remain source-reported; raw pilot outputs and the complete fresh-probe execution history are not included. This revision adds no new study execution or human-reference validation result.

Files

semantic-conformance-report-v1-2.pdf

Files (59.8 kB)

Name Size
md5:bd5bba6b35b7244c0aaead13683fab45
59.8 kB Preview Download

Additional details

Additional titles

Subtitle
A case study of authored calibration and fresh audit failures

References

  • Jacobs, A. Z., & Wallach, H. (2021). Measurement and Fairness. FAccT 2021, 375–385. https://doi.org/10.1145/3442188.3445901
  • Raji, I. D., Bender, E. M., Paullada, A., Denton, E., & Hanna, A. (2021). AI and the Everything in the Whole Wide World Benchmark. NeurIPS Datasets and Benchmarks 2021. arXiv:2111.15366. arXiv DOI: https://doi.org/10.48550/arXiv.2111.15366
  • Ribeiro, M. T., Wu, T., Guestrin, C., & Singh, S. (2020). Beyond Accuracy: Behavioral Testing of NLP Models with CheckList. ACL 2020, 4902–4912. https://doi.org/10.18653/v1/2020.acl-main.442
  • Zheng, L., et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS Datasets and Benchmarks 2023. arXiv:2306.05685v4. arXiv DOI: https://doi.org/10.48550/arXiv.2306.05685
  • Chapman, W. W., Bridewell, W., Hanbury, P., Cooper, G. F., & Buchanan, B. G. (2001). A simple algorithm for identifying negated findings and diseases in discharge summaries. Journal of Biomedical Informatics, 34(5), 301–310. https://doi.org/10.1006/jbin.2001.1029
  • Harkema, H., Dowling, J. N., Thornblade, T., & Chapman, W. W. (2009). ConText: An algorithm for determining negation, experiencer, and temporal status from clinical reports. Journal of Biomedical Informatics, 42(5), 839–851. https://doi.org/10.1016/j.jbi.2009.05.002