Replication Package for "Position: Current Coding Benchmarks Measure Task Performance, not Coding Capabilities"

This replication package accompanies the paper "Position: Current Coding Benchmarks Measure Task Performance, not Coding Capabilities" and contains:

Evaluation Results for 24 models across 3-4 benchmarks:

• Method Generation — Multi-turn code generation with real-world test feedback from the Django framework
• Code Completion — Single-turn completion tasks
• Program Repair — Bug fixing with test-driven feedback
• LiveCodeBench — Competitive programming evaluation

Models Evaluated:

• 10 open-weight models (Qwen2.5-Coder, DeepSWE, Skywork-SWE, R2EGym, SWE-agent-LM, OpenHands)
• 6 LoRA fine-tuned configurations (32B and 7B variants)
• 8 proprietary API models (GPT-4.1, GPT-5, Claude Haiku/Sonnet/Opus 4.5, Gemini 2.5/3)

Fine-tuned Checkpoints:

• LoRA adapter weights for Qwen2.5-Coder-32B-Instruct and Qwen2.5-Coder-7B-Instruct. Fine-tuned on method generation, completion, and program repair tasks

Analysis Tools:

• Python utilities for loading and analyzing results
• Jupyter notebook for interactive exploration