Compression Is Intelligence : Rethinking Neural Network Generalization

This research note studies how neural networks select among the many functions that are consistent with a finite training set. On enumerable input domains, we record each network’s predictions over the complete domain and show that function selection continues after every training example has already been classified correctly. Actual training therefore cannot be fully described as a one-time conditioning of the initialization function prior on training-label consistency.

By sampling parameters directly at progressively lower training-loss thresholds, we find that the parameter probabilities of different functions shrink at different rates and can even reverse their ordering. We summarize this behavior with a loss-resolved Neural Kolmogorov complexity curve. In controlled experiments, these curves predict the number of samples required to recover a rule, candidate labels for unseen examples, and function frequencies produced by repeated training.

The framework distinguishes complete-target volume, the conditional function distribution under a fixed training set, and the distribution actually reached by an optimizer. Experiments span enumerable Boolean rules, Rule110, grokking, and whole-network HMC on 50,000 MNIST training images. The central interpretation is that reducing training loss simultaneously eliminates functions inconsistent with the data and increases the relative probability of functions that reuse structure more efficiently.

Code, experiment records, and reproduction materials: https://github.com/ball-lightning6/neural-sculpting-paradigm