Published July 13, 2026 | Version v1

A world model as a judge: early, auditable verdicts on partial robot-manipulation rollouts.

Description

We present LeWorldModel Judge, a locked benchmark and an auditable judging surface for scoring partial robot-manipulation rollouts before their episodes end.

Sparse terminal reward answers no useful question about a trajectory prefix: whether the rollout is still on track, already doomed, or physically implausible only becomes visible in hindsight, which makes early triage, data curation, and process-reward experiments needlessly blind.

We lock a narrow evaluation slice (three Meta-World tasks, four constructed policy families, prefix cutoffs at 0.25, 0.50, and 0.75 of the horizon), define heuristic but fully inspectable prefix labels, and score every prefix with a cutoff-time composite judge over named evidence signals, alongside a replay-time latent proxy whose causal limitation we state exactly.

A mandatory calibration protocol reports the provenance of every decision threshold and only grants held-out status when the calibration and evaluation policy families are disjoint. On a fresh 90-prefix held-out slice collected after the label rules were frozen, the composite judge reaches a 0.949 failure hit rate at a 0.137 false positive rate and 0.978 pairwise ranking accuracy, against 0.50 and 0.554 for sparse-reward and progress baselines, and the threshold calibrated on disjoint families transfers across disjoint captures to within 0.0004 (0.2977 versus 0.2980). Every reported number decomposes to named sub-scores in checked-in files, carries a judge_mode provenance tag, and regenerates from a pinned command line.

Files

leworldmodel-judge.pdf

Files (241.5 kB)

Name Size
md5:4211a6bab5f88c2184f1d6c01f2409e8
241.5 kB Preview Download

Additional details

Software