PreCap: A Bounded, Self-Normalizing Activation Function that Matches SELU in Deep Normalization-Free MLPs

We introduce PreCap, a smooth, monotonic activation function that is bounded on both sides: an exponential unit that switches to its tangent line above a threshold (CapExp, which is exactly a shifted and scaled ELU) is applied after a soft tanh clip. SPreCap scales PreCap so that, like SELU, zero mean and unit variance are a fixed point, and adds an input gain b that sets the mean-field gradient gain χ1. In a preregistered study on 50- and 100-layer MLPs without normalization or skip connections (MNIST and FashionMNIST, per-activation learning-rate tuning, 15 confirmation seeds, Holm correction), SPreCap-cs with χ1 equal to SELU matches SELU: no cell is significantly worse, with median paired differences between -0.12 and +0.20 percentage points. Test error decreases monotonically as χ1 approaches 1 in all four cells. The variant SPreCap-e03 (χ1 = 1.03) beats SELU on MNIST at depth 100 (5.28 % vs. 6.36 %, Holm-adjusted p = 0.0064, secondary test), while plain ELU is about as strong overall. PreCap is not more robust to the learning rate. Earlier large advantages at a fixed learning rate disappeared under fair tuning, and we report these negative results. We release fused CUDA and Apple MLX kernels that run within about 1.5× of native SELU in float32 and an installable package.

Code, data and kernels: https://github.com/hermi-labs/precap-activation