AI Coding Proficiency

We conduct the first comprehensive empirical study examining AI proficiency across 170 third-party libraries and 61 task scenarios, evaluating six widely used LLMs. Our findings reveal that libraries with similar functionalities can exhibit up to 84% differences in the quality score of LLM-generated code, while different models also exhibit quality gaps among their generation results using the same library

This repository contains the prototype of both the dataset construction pipeline and the evaluation scripts, as well as our datasets and analysis results.