Binary Saturation Dynamics in Recurrent Spiking Neural Networks: Limitations of Homeostatic Regularization

Published February 27, 2026 | Version v1

Journal Open

The recent evolution in Neural Networks has led to the introduction of a new

promising Network known as the Spiking Neural Network (SNNs) which is supposed

to be energy-efficient due to the fact that it only spikes fire over time. Practically

when these networks are trained with surrogate gradients, their activities often

become unstable, either almost all neurons fire too much (“epileptic saturation”) or

almost none fire at all (“comatose silence”) due to non-differentiable nature of spike

generation. In this study we investigate the efficacy of Homeostatic L2 Regularization

in enforcing sparsity constrains on a rate-coded RSNN for temporal anomaly

detection. Homeostatic L2 regularization technique pushes the average firing rate

towards desired target of 5% and theoretical this should keep the RSNN sparse.

However, the experiment shows otherwise and the network settles into a high-actively

regime with about 49.2% of neurons firing while it achieves a 100% classification

accuracy on the task, which implies that the network solves the problem but in a very

dense, power-hungry way, more like the conventional digital circuit than a sparse

brain-like SNN.

However, it is suggested that the gradient descent landscape for rate-coded RSNNs

prioritizes noise robustness (maximum signal-to-noise ratio via saturation) over

metabolic efficiency, rendering soft regularization insufficient. Such that the

“Synaptic Efficiency” metric was adopted to quantify how much useful computation

or information transfer is achieved per unit of the synaptic activity. With this metric it

fell out that networks can be accurate yet synaptically inefficient when they operate in

saturated regimes. Based on this observation it was concluded that achieving truly

neuromorphic sparity cannot relay less-functional penalties alone. Instead, the

architecture must include hard constraints like enforced refractory periods, strict

limits on firing, or other built-in mechanisms so as to structurally prevent the network

from drifting into high-firing, energy-expensive modes while training.

Files

Name	Size	Download all
Binary Saturation Dynamics in Recurrent Spiking Ne.pdf md5:0726be14294ea6fb6c2a91bff3bdd1c4	345.4 kB	Preview Download