QuAC: Quick Attribute-Centric Type Inference for Python

This artifact supports the paper "QuAC: Quick Attribute-Centric Type Inference for Python." The artifact includes the code and data used to generate the results presented in the paper. It aims to reproduce the main scientific claims and facilitate future research by making the software publicly available.

This is a list of the initial results in the manuscript:

- Evaluating **Stray, HiTyper, and QuAC** on **untyped benchmarks** using the **correctness modulo type checker** approach.
  - Table 2. The total number of non-trivial (i.e., not None or typing.Any) type predictions by each technique on each benchmark.
  - Table 3. Total number of container type predictions with non-trivial type parameters (i.e., list[int] rather than list) by each technique.
  - Table 4. Percent of errorless type predictions that are non-builtin types (left); total number of errorless non-builtin type predictions (right).
  - Fig. 2. Typing slots on which each technique makes errorless non-trivial type predictions over all benchmarks.
  - Fig. 3. Intersection over union between typing slots with errorless non-trivial type predictions, per pair of techniques and benchmark.
  - Table 5. Run times of each technique in seconds.
  - Fig. 4. Log scale run time of each technique (y-axis) plotted against lines of code in each benchmark (x-axis).
  - Table 6. Failure modes of each technique.

This artifact supports all claims *EXCEPT* "Table 6. Failure modes of each technique," which requires a manual analysis - we took the five most error-prone typing slots for each method on each benchmark, and manually categorized the reasons for type prediction failure.

For Table 5 (and accordingly Figure 4), we have notices a discrepancy in QuAC's runtime data after submission, which we have uploaded with our R2 response. Fresh-baked results should be similar to: 

| Repository Name | Stray | HiTyper | QuAC |
| --- | --- | --- | --- |
| requests | 80.7 | 48.5 | 10.4 |
| Pygments | 2606.1 | 365.4 | 62.9 |
| boto3 | 28534 | 79.2 | 7.83 |
| gunicorn | 193.3 | 81.5 | 22.9 |
| python-dateutil | 270.9 | 154.8 | 19.6 |
| pytz | 38.1 | 24.3 | 2.81 |
| six | 6.57 | 1.94 | 1.54 |
| pytest-cov | 38.2 | 9.85 | 1.96 |
| notebook | 13.3 | 9.38 | 1.57 |
| peewee | 3.15 | 2.09 | 20.2 |
| seaborn | 5400.0 | 259.4 | 193.9 |

*Note that Stray and QuAC produce slightly different results from run to run due to selecting different top-ranking type annotations. The runtimes of the methods also vary from run to run. However, the general trends remain.*