Artifact of the paper: Tutoring LLM into a Better CUDA Optimizer

This repository contains the replication package for the paper titled "Tutoring LLM into a Better CUDA Optimizer" presented at Euro-Par 2025. The package contains the source files and supplementary scripts for a testing framework for evaluating LLM-generated computation kernels for three assignments: Game of Life simulation, histogram computation, and the k-Nearest Neighbors search. The package also contains all the collected measurements, the used LLM prompts and generated responses, the interactive scenarios presented in the paper, and all analyses conducted by the researchers.

 

The file structure of the package and further details are in the included README.md file. The package assumes a CUDA-accelerated Linux platform with GCC 13.2 or higher, NVCC 12.6 or higher, CMake 3.20 or higher, and Python 3.8 or higher.

 

The default configuration of LLM prompting is set up for GPT-o3-mini, which was the state-of-the-art model for coding tasks at the time of writing; however, the package is prepared for reproducibility on more recent models. Collecting LLM-generated implementations and their subsequent evaluation are fully automated.

 

For a quick confirmation that the target platform is prepared for the evaluation, enter the following shell commands:

# Depending on the LLM assignment you want to test, one of the following:
cd framework/histogram              # Histogram base directory
cd framework/game-of-life/infrastructure  # Game of Life base directory
cd framework/knn                   # k-NN base directory

make      # Compiling the code
make run  # Example: run the baseline implementation

The commands should output the mean time and standard deviation of the recordings and the validation result on baseline implementations. To re-evaluate the implementations generated by the GPT-o3-mini model, follow the instructions in the included README.md file (section Replication).

To reproduce the graphs presented in the paper, run the following shell commands (does not require re-evaluating the implementations):

cd measured-times
bash generate_all_graphs.sh