Reusing Model Probabilities for New Queries

To predict a coarse category, a classifier can either estimate the category's probability directly or sum the predicted probabilities of its fine-grained constituent labels. While probability aggregation guarantees consistency across different label groupings by construction, whether it also improves predictive accuracy remains an empirical question. We investigate this trade-off for out-of-domain intent classification using domains excluded from fine-tuning. For a zero-shot classifier, aggregating fine-grained probabilities lowers the Brier score from 0.95 to 0.60. For two prompted language models, however, the benefit of aggregation depends heavily on prompt design: it is substantial when coarse categories are specified only by short labels, but largely vanishes when prompts explicitly list each category's underlying intents. During fine-tuning, introducing a consistency penalty aligns direct predictions with aggregated ones and substantially improves direct estimation, though it yields no gains for aggregated prediction itself. Conversely, light fine-tuning with hyperparameters tailored for aggregation improves Brier scores regardless of the penalty, albeit at the expense of classification accuracy. From an efficiency standpoint, reusing a single fine-grained distribution was faster than querying each grouping directly in the measured workloads requiring two or more groupings for the same input. Furthermore, controlled synthetic experiments demonstrate that answering logical queries from a joint distribution generalizes well to unseen conjunctions, while a companion semantic-parsing study reveals that learned parsers frequently generate syntactically valid programs with incorrect semantics—even on descriptions that humans interpret without error. We conclude that consistency by construction serves as a strong baseline whose practical value depends on prompt informativeness, the training regimen, and the query workload.