Published June 21, 2026 | Version v1

Measuring Context Gain over Endings-Only Shortcuts in a Human-Written Narrative Cloze Task

Authors/Creators

Description

Narrative cloze tasks are widely used to study whether a model understands a story well enough to pick a coherent continuation. A recurring problem is that such tasks can be partly solved without reading the story at all: incorrect endings often differ from correct ones in style or wording, so a model can succeed by reading the ending alone. This work treats that shortcut not as a nuisance to be hidden, but as the quantity to measure.

We build an automatic narrative cloze task from public ROCStories-style data in which both endings are genuinely human-written: the wrong ending is retrieved from a different human story by nearest-neighbor similarity, rather than generated from a template. On this task we report three transparent baselines — an endings-only classifier, a context-plus-ending classifier, and a compressed bag-of-words representation — and measure each one's gain over the endings-only shortcut.

Across four random seeds (20,000 examples each), the endings-only classifier already reaches 58.9%, confirming the shortcut is real. The context-plus-ending model adds +5.2 points and the compressed representation adds +8.0 points over it. Under an entity-masking stress test that removes name cues, both gains shrink to roughly +3.9 points, showing that a meaningful part of the apparent "context understanding" is entity-string overlap.

We make no claim of causal or world-model reasoning. The contribution is a reproducible, shortcut-controlled evaluation protocol and an honest accounting of how much measured context gain survives when surface cues are removed.

Files

Merrout_NarrativeCloze_paper.pdf

Files (124.3 kB)

Name Size
md5:cb772a620bca1f9d7c6ff222fe74b461
124.3 kB Preview Download