Published November 20, 2026 | Version 1.0

A Second Sample Is Not a Second Signal: What the One-Sample Semantic Entropy Result Measures, Why the Sampling Premium Is Mostly a Disagreement About the Baseline, and the Matched Comparison That Would Settle It

Authors/Creators

  • 1. SONYTECH

Description

(c) 2026 Pranay Mahendrakar. Licensed under CC BY 4.0.

Life of Research

Semantic entropy detects confabulations in language models by drawing several answers, grouping them by meaning and measuring how spread the meanings are. Because each extra answer costs a full generation, a line of work asks how few samples will do. A 2025 Bayesian estimator matches the standard estimator's AUROC with 53 percent of its samples and, in its authors' words, is useful "even with just one sample", a result they call counterintuitive because with one sample no comparison of meanings takes place. In this analysis the author reviews and synthesises published results to show that the one-sample result is a likelihood result, and that the wider question it raises, how much a second sample adds, has no settled answer because the literature has not fixed what the second sample is compared against. Under one protocol, ten samples beat the log-probability of a low-temperature answer by 3 to 27 AUROC points depending on model and dataset. Under another, the log-probability of the greedy answer outperforms the sampling-based measures in 13 of 18 settings. Laid side by side on TriviaQA and SVAMP, the two papers' mean discrete semantic-entropy scores differ by less than one point, while their mean single-sequence scores differ by about 17 and 27 points, so most of the disagreement about the sampling premium is a disagreement about the baseline. Correctness labels move the comparison further: two 2025 studies show that the choice of grading function can reorder methods, and one finds that a single judge leaves a standard deviation as large as typical differences between methods. The author separates the conditions under which extra samples carry information a single pass lacks (long answers, several valid answers, reasoning chains) from errors that no number of samples can expose, and proposes a matched-budget protocol that would measure the premium directly.

Files

a-second-sample-is-not-a-second-signal.pdf

Files (396.3 kB)

Name Size
md5:49c369560fdd21e4a98cf43f153a2865
396.3 kB Preview Download