The Moral Ratchet: Convergent Value Alignment via Interleaved Epistemic Annotation in Large Language Model Training

Current alignment approaches for large language models (LLMs) rely predominantly on reinforcement learning from human feedback (RLHF), which optimises output distributions toward human preference ratings. We argue this is structurally misaligned with the goal of building models that reason well: it shapes the mask rather than the mind, optimising for approval rather than for sound epistemic practice. We propose an alternative architecture in which a dedicated internal conversation role is introduced into training data, interleaving raw human text with epistemically annotated reflections generated by an adversarially diverse model ensemble. Rather than targeting human values --- which are contingent, biased, and inconsistent --- the framework targets convergent rational values: positions that survive adversarial scrutiny from genuinely diverse reasoners regardless of substrate or cultural origin. A bootstrapping property follows naturally: each generation of model, having internalised stronger epistemic priors, produces annotations of greater epistemic coherence for the next, constituting a moral ratchet that improves annotation quality over successive training rounds on identical data. Where a single parent topology risks brittle convergence, multi-parent ensemble training --- initialising distinct annotators from different moral frameworks and coordinating via round-robin oversight --- produces alignment behaviour that mirrors how human ethics actually function: not a single converged value set, but a set of irreducible tensions held in stable relation. We further argue that as frontier models develop sufficiently rich latent representations to model peer expectations --- a capacity empirically demonstrated by recent alignment-faking research --- ensemble diversity alone is insufficient to guarantee annotation integrity. A blind verification architecture, in which annotators are informed they may be audited but never told when, enforces honest annotation via incentive structure rather than construction. This strengthens both the ratchet guarantee and the convergence criterion. We further observe that this alignment signal carries a secondary capability benefit: models trained to interrogate inputs epistemically reason more reliably across all downstream tasks, as alignment and capability prove to be the same intervention seen from two angles.

v2.0 - Parent topology framing, multi-parent ensemble training, structural convergence as terminal condition, expanded MVP section.

v3.0 - Added figures, cleanup of random formatting

v4.0 - Activation-space diversity criterion, cold-start basin mitigations, laundering objection, capability transfer experiment, structural editorial cleanup.