CompToolBench: A Compositional Benchmark for Fine-Grained Tool-Use Evaluation of Large Language Models

We introduce CompToolBench, a benchmark for evaluating LLM tool-use across four composition levels (single, sequential, parallel, and graph) using 200 tasks and 106 real tools drawn from free public APIs. We evaluate 18 models spanning cloud and local deployments and identify a Selection Gap: models that correctly select tools frequently fail to produce valid calls, with an average gap of 13.2 percentage points. Results show that compositional complexity does not uniformly degrade performance, and that local models now approach cloud-model accuracy on structured tool-use tasks.