Beyond Static Bias: A Case for Dynamic, Per-Neuron Adaptation in Deep Networks

The process of feature learning in deep networks often
appears random and un-interpretable. To introduce
a more structured approach, we propose a method
to achieve ”Less Randomness” in neural computation through Adaptive Bias Networks (ABN). ABNs
replace the monolithic, static bias of a layer with
dynamic, per-neuron modulation mechanisms. Each
neuron learns to ”reason” about the input by generating a context-specific signal that adaptively reweights its own effective parameters. This is achieved
via a gating mechanism that consults a bank of specialized bias vectors. When integrated into a standard ResNet on the CIFAR-10 benchmark, our ABN
model demonstrates a consistent performance improvement over the baseline. This suggests that empowering individual neurons with adaptive reasoning
capabilities is a more efficient and powerful way to
structure model parameters, leading to less random
and more effective learning.