The Reasoning Efficiency Frontier: Compute-Aware Evaluation of Accuracy, Cost, and Latency in Frontier Reasoning Models
Authors/Creators
Description
Frontier language models increasingly use test-time computation to improve performance on complex
reasoning tasks. This shift has changed the problem of model evaluation: the most accurate model is not
necessarily the most efficient model, and maximum reasoning effort is not necessarily the optimal
operating point.
Existing research has established several important phenomena, including overthinking on simple tasks,
diminishing returns from additional reasoning, and the need for adaptive compute allocation. However,
model evaluations are often conducted within a single benchmark, use model-specific metrics, or focus
primarily on accuracy and token efficiency.
This paper proposes a reproducible evaluation framework for measuring the reasoning efficiency frontier
of frontier language models across three jointly optimized dimensions: accuracy, inference cost, and
latency. The framework evaluates models across task families and empirically defined difficulty levels
while varying provider-native reasoning effort. For each inference, the framework records correctness,
token consumption, reasoning effort, cost, time to first token, end-to-end latency, and output reliability.
Rather than producing a single model leaderboard, the framework constructs accuracy–cost–latency
Pareto frontiers and estimates the marginal value of additional reasoning compute. We further propose a
workload-level evaluation in which a fixed inference budget is allocated either uniformly or according to
task difficulty. To motivate the framework, we summarize convergent evidence from four independently
published benchmarks — LLMThinkBench, OptimalThinkingBench, When More Thinking Hurts, and Plan
and Budget — which together show non-monotonic and sometimes negative relationships between
reasoning length and accuracy.
The resulting framework is intended to answer a practical question that conventional benchmarks do not
fully address: how much thinking is enough for a given task, model, and workload constraint? The work
provides an experimental protocol, evaluation metrics, reproducibility requirements, and an
implementation framework suitable for evaluating rapidly changing frontier reasoning models. Consistent
with the study-protocol format, this manuscript specifies the framework, metrics, and hypotheses ahead
of data collection; the pilot and full-scale runs described in Section 18 have not yet been executed, and
first-party experimental results will be reported in a subsequent publication.