Controlled Language Models Inference-Time Control, Tokenization Engineering, and Reversible Optimization

This work presents a complete, reproducible framework for controlling large language models without relying on repeated fine-tuning or RLHF. It reframes language models as controlled dynamical systems whose behavior, efficiency, and stability are governed at inference time through predictive state monitoring, reversible optimization, and adaptive tokenization.

The core validated contribution is the discovery that lightweight classifiers trained on transformer hidden states can predict degenerative behaviors—most notably repetition—with extreme separation (125×) before those behaviors manifest in output tokens. This enables proactive decode-time intervention rather than reactive filtering or retraining.

The framework further introduces tokenization engineering as a first-class control surface. Tokenization is treated not as a fixed preprocessing step, but as a co-evolving interface whose structure directly shapes model efficiency, context utilization, and stability. Diagnostic signals and commit/rollback semantics enable safe tokenizer evolution without catastrophic regressions.

A staged dense-response training pipeline addresses RLHF-induced verbosity, while a bounded recursive optimization loop demonstrates stable, reversible self-optimization under frozen evaluation criteria. Negative results, failure modes, and training regressions are explicitly documented.

This release consolidates architectural specifications, training pipelines, evaluation methodology, tokenizer diagnostics, implementation guidance, and reproducibility constraints into a single authoritative reference. All claims are clearly labeled as validated, bounded, or theoretical.

The work does not claim AGI, open-ended self-improvement, or autonomous operation. Its contribution is architectural: demonstrating that many persistent language-model failure modes are control problems rather than training problems—and that they can be solved with principled systems design on consumer hardware.

Released under CC BY 4.0 to support verification, replication, and extension by the research community.