Novice Type Error Diagnosis with Natural Language Models - Artifacts

Strong type system helps programmers to eliminate many er-
rors without much burden of supplying type annotations. However, this
flexibility makes it highly non-trivial to diagnose the failure mode, es-
pecially for novice programmers. Compared to classic constraint solving
and optimization-based approaches, the data-driven approach has shown
great promise in identifying the root cause of type errors with high accu-
racy. Instead of relying on hand-engineered features, this work explores
natural language models for type error localization, which can be trained
in an end-to-end fashion without requiring any features. We demonstrate
that, for novice type error diagnosis, language model based approach
significantly outperforms the previous state-of-the-art data-driven ap-
proach. Specifically, our transformer model could predict type errors
correctly 62% of the time, outperforming the state-of-the-art Nate’s
data-driven model by 11%, in a more rigorous metric of accuracy mea-
surement. Furthermore, we also apply structural probe to explain the
performance difference of different language models.

Artifacts

• Kaggle Notebooks
  1. inference-7fa4ae-final.ipynb
  2. inference-for-model-training.ipynb
• nater.ova (Virtual Machine)
  1. Scripts for computing NATE models's accuracies under the new metric
• Scripts for Generating Graphs
  1. Accuracy of Type Error Localization Techniques.py
  2. Impact of type error threshold on blame accuracy.py
  3. Impacts on training loss.py

The size of this partifact package is about 5.2 Gb