Isotropic Latent Perturbation is Just Weight Decay: A Critical Deconstruction of Stochastic Feature Blurring

Over the past several years, isotropic stochastic perturbation of multi-dimensional latent spaces has been widely celebrated as a foundational breakthrough for mitigation of overfitting.

In this work, we will present a rigorous mathematical deconstruction that strips away the surrounding rhetoric. Through formal Taylor expansion, we will attempt to prove that injecting spherical isotropic noise into a network's hidden layers is mathematically equivalent to a dynamically scaled L₂ regularization (weight decay) on the preceding layer's weight matrix.