How much does an LLM know about my programming language?

This repository is the official artifact for the SLE 2026 paper "How much does an LLM know about my programming language?".

The framework systematically evaluates LLM proficiency in a programming language using ANTLR-based syntactic analysis: it extracts keywords and grammar rules from LLM-generated code and measures how much of the language's vocabulary and structure a model actually uses — even in code that fails to compile.

Key features:

• Granular metrics: Measures Keyword/Rule Coverage and Production Validity.
• Partial knowledge capture: Analyzes syntactically invalid code to identify where a model's knowledge fails.
• Extensible: Designed to work with any language that has an ANTLR grammar.
• Automatic reporting: Generates a complete language coverage report as a Jupyter Notebook.