TypeEvalPy: A Micro-benchmarking Framework for Python Type Inference Tools

GitHub link: https://github.com/secure-software-engineering/TypeEvalPy

A Micro-benchmarking Framework for Python Type Inference Tools

📌 Features:

• 📜 Contains 154 code snippets to test and benchmark.
• 🏷 Offers 845 type annotations across a diverse set of Python functionalities.
• 📂 Organized into 18 distinct categories targeting various Python features.
• 🚢 Seamlessly manages the execution of containerized tools.
• 🔄 Efficiently transforms inferred types into a standardized format.
• 📊 Automatically produces meaningful metrics for in-depth assessment and comparison.

🛠️ Supported Tools

🏆 TypeEvalPy Leaderboard

Below is a comparison showcasing exact matches across different tools, coupled with top_n predictions for ML-based tools.

_{(Auto-generated based on the the analysis run on 20-10-23 14:51)}

🐳 Running with Docker

1️⃣ Clone the repo

git clone https://github.com/ashwinprasadme/TypeEvalPy.git

2️⃣ Build Docker image

docker build -t typeevalpy .

3️⃣ Run TypeEvalPy

🕒 Takes about 30mins on first run to build Docker containers.

📂 Results will be generated in the results folder within the root directory of the repository. Each results folder will have a timestamp, allowing you to easily track and compare different runs.

docker run \
      -v /var/run/docker.sock:/var/run/docker.sock \
      -v ./results:/app/results \
      typeevalpy

🔧 Optionally, run analysis on specific tools:

docker run \
      -v /var/run/docker.sock:/var/run/docker.sock \
      -v ./results:/app/results \
      typeevalpy --runners headergen scalpel

🛠️ Available options: headergen, pyright, scalpel, jedi, hityper, type4py, hityperdl