Published October 16, 2026 | Version 1.0

Learning from the Learner's Response: Bi-Objective Acquisition for Sparse Expert Supervision in Reinforcement Learning

Authors/Creators

  • 1. ROR icon Arts et Métiers

Description

Sparse expert supervision can improve reinforcement learning, but deciding which
states are worth annotating remains difficult. Uncertainty identifies where a learner
is unsure, but does not directly identify where supervision will most strongly
change that learner. We introduce ALBERTA, a response-aware acquisition frame-
work that measures local policy and value changes produced by controlled expert
interventions, predicts those responses from pre-annotation signals, and selects
candidates in the resulting two-dimensional response space. A calibration set of
200 expert-labeled probes is used to learn response from policy entropy, historical
critic disagreement, and their interaction; Pareto selection then allocates a separate
121-state downstream budget. The response-aware branch therefore uses 321 ex-
pert labels in total, compared with 121 for the Random baseline. In a controlled
single-seed Atomic Chess experiment from a common RL10 branch point, the
Pareto branch achieves a higher mean game score than PPO-only at all four tested
evaluation temperatures, with smaller margins at higher temperatures, and shows
its largest margins over Random at lower temperatures. These results provide
evidence that learner response can be a useful acquisition target while exposing
substantial differences between policy and value responsiveness.

Files

preprint_atomic_chess.pdf

Files (1.4 MB)

Name Size
md5:c549f088c01810b8b4c9afc566a55917
1.4 MB Preview Download

Additional details

Identifiers

Dates

Available
2026-10-16