Continual Learning via Post-Hoc Weight Merging: without catastrophic forgetting

This paper introduces a lightweight method for continual learning in neural networks — the ability to learn new classes one at a time without forgetting previously learned ones — without replay buffers, regularization penalties (e.g. EWC), or growing the network architecture.

The core idea: a disposable "learner" network is trained on one class at a time, then its weights are merged directly into a persistent "accumulator" network through simple, index-wise addition. A layer-wise normalization scheme keeps the accumulator's weights from saturating as more classes are added over time.

Tested on a full ten-class sequential MNIST digit classification task, the method reaches a mean accuracy of 76.4% across three independent runs, with strong retention (80–99%) on most digits and a specific, well-characterized weakness on visually overlapping classes (5 and 9), traced through direct pairwise testing to representational overlap between those digits rather than a general failure of the approach.

This is an honest, in-progress account of the method, including preliminary experiments and open questions — such as an under-investigation "remerging" refinement step — rather than a fully polished result. Both what worked and what didn't are reported throughout.