Artefact for Hallucination Inspector: A Fact-Checking Judge for API Migration

This repository contains the official artefact and source code for the paper Hallucination Inspector: A Fact-Checking Judge for API Migration, presented at EASE 2026

The Hallucination Inspector is a lightweight, documentation-guided static analysis framework designed to detect a specific failure mode in Large Language Models (LLMs) called Scaffolding Hallucination. This occurs when models generate incorrect calling contexts for APIs, such as using imaginary imports, constructors, or constants, which we term Phantom Symbols

Included Materials

• Source Code: The Python implementation of the Hallucination Inspector, utilising the Tree-sitter library for AST parsing and symbol extraction.

   

• API Oracle: A ground-truth symbol table derived from the official Android API specification (XML format), flatten recursively to support efficient membership checks for classes, constants, and methods.

• Evaluation Dataset: A curated dataset of 51 Android API migration pairs used to validate the tool's efficacy against standard metrics like CodeBLEU and LLM-as-a-Judge baselines.

• Experiment Scripts: Tools to reproduce the precision and recall metrics reported in the paper, where the Hallucination Inspector demonstrated superior performance by grounding its verification in a deterministic oracle

   

For further technical details, please refer to the full paper or the documentation included within this archive.