DataKernelBench

Can LLMs optimize database queries on GPUs?

DataKernelBench evaluates LLMs on a novel task: optimizing analytical database queries as GPU kernels. It first represents each SQL query as a validated PyTorch program called a TorchPlan. It then evaluates LLMs by asking them to optimize either the tensor-intensive core (core) or the full query implementation (full) using CUDA or Triton, with execution-guided repair. The benchmark covers all 22 TPC-H queries.

On TPC-H SF10 using one NVIDIA H100, the strongest configuration—GPT-5.5 with CUDA at the full-query level—achieves 2.11× overall speedup over compiled TorchPlan with a 100% pass rate. The paper evaluates ten proprietary and open-weight models and analyzes how model strength, backend, optimization scope, and prompt context affect correctness and performance. 

Links: Project page · Paper · Code · Data: Hugging Face · Zenodo