Learning from the Learner's Response: Bi-Objective Acquisition for Sparse Expert Supervision in Reinforcement Learning
Description
Sparse expert supervision can improve reinforcement learning, but deciding which
states are worth annotating remains difficult. Uncertainty identifies where a learner
is unsure, but does not directly identify where supervision will most strongly
change that learner. We introduce ALBERTA, a response-aware acquisition frame-
work that measures local policy and value changes produced by controlled expert
interventions, predicts those responses from pre-annotation signals, and selects
candidates in the resulting two-dimensional response space. A calibration set of
200 expert-labeled probes is used to learn response from policy entropy, historical
critic disagreement, and their interaction; Pareto selection then allocates a separate
121-state downstream budget. The response-aware branch therefore uses 321 ex-
pert labels in total, compared with 121 for the Random baseline. In a controlled
single-seed Atomic Chess experiment from a common RL10 branch point, the
Pareto branch achieves a higher mean game score than PPO-only at all four tested
evaluation temperatures, with smaller margins at higher temperatures, and shows
its largest margins over Random at lower temperatures. These results provide
evidence that learner response can be a useful acquisition target while exposing
substantial differences between policy and value responsiveness.
Files
preprint_atomic_chess.pdf
Additional details
Identifiers
Dates
- Available
-
2026-10-16