MemGuard: Predicting the Post-Optimal Training Regime in Large Language Models

This work analyzes training dynamics in large language models and identifies a critical training step tc, after which continued optimization degrades generalization.

We analyze multiple publicly available LLM training runs (including Zephyr-7B, Mistral-7B and related models) and observe three empirical laws:

• Post-optimal degradation scales with the overshoot ratio

drift ∝ log(last_step / tc)

• The optimal step tc follows scaling consistent with model size

tc ~ N log N

• Dataset informational complexity K(D) correlates with tc

Based on these observations we introduce MemGuard, a lightweight monitoring algorithm detecting the onset of the post-optimal regime during training.

Empirical analysis suggests that stopping training near this transition can reduce unnecessary compute while preserving model performance.

This repository contains the paper and figures illustrating the phenomenon.

Intellectual Property Notice:

This work has been deposited as a prior disclosure through an INPI Enveloppe Soleau (France), dated 08 March 2026.