PALA: Plateau-Aware Learning with Adaptive Memory Faster Convergence via SNR-Guided Plateau Classification and Cosine Bounce Restarts for Adam-Family Optimizers

PALA (Plateau-Aware Learning with Adaptive Memory) matches ReduceLROnPlateau in final accuracy
while using 25% fewer training epochs, reaching competitive accuracy 13 epochs sooner and early-stopping at
a mean of 75 epochs versus 100 for all fixed-schedule baselines. This convergence efficiency is PALA’s primary
practical contribution: the same validation accuracy for a quarter less GPU compute, with no schedule-specific
hyperparameter tuning.
PALA achieves this by diagnosing why a model has stalled, not merely that it has stalled, and responding
with a geometry-adapted escape strategy. It reads the first and second Adam moments already maintained by
the optimizer at zero additional cost: the gradient signal-to-noise ratio SNR = ∥E[g]∥/∥E[g2]1/2∥ and the
gradient variance proxy GVar = E[∥v1/2
t ∥] are jointly mapped through a threshold classifier to one of three
landscape categories (SADDLE, FLAT, or SHARP), each triggering a type-specific cosine learning-rate bounce. A
region-of-interest (ROI) memory accumulates (floor_lr, loss_gain, plateau_type) triples across
bounces; the next floor is selected by a recency–ROI weighted mean over type-matched entries. Adaptive β2
scaling accelerates second-moment adaptation during bounces, and a β1 reduction on saddle exits increases
gradient direction reactivity. A gradient-agreement detector escalates bounce amplitude when the optimizer
re-enters the same trap. On CIFAR-100 (PalaCNN) and STL-10, PALA outperforms plain Adam by +3.8 pp on
average and matches ReduceLROnPlateau to within 0.2 pp in final accuracy. Our ablation further reveals that
a more aggressive floor decay (γ=0.1) yields 63.3%, surpassing ReduceLROnPlateau (62.0%) by 1.3 pp.
On ResNet-18, PALA underperforms in the 100-epoch regime due to BatchNorm’s suppression of gradient
variance signals.