There is a newer version of the record available.

Published June 6, 2026 | Version v2
Report Open

Born Again ReLU (BAReLU): A Branchless Multi-Regime Fusion Activation Function for Resilient Gradient Propagation and Saturated Signal Decoupling

Authors/Creators

Description

Born Again ReLU (BAReLU): A Branchless Multi-Regime Fusion Activation Function for Resilient Gradient Propagation and Saturated Signal Decoupling

Author: Anthony Aseervatham (Independent Researcher)

Overview

This repository contains the research paper, experimental data, and supporting figures for BAReLU (Born Again ReLU), a novel activation function designed to eliminate the "Dying ReLU" problem while maintaining zero computational overhead.

The Problem

Standard Rectified Linear Units (ReLU) suffer from a critical flaw known as the Dying ReLU epidemic: during training, neurons can enter permanently inactive states where both forward activations and backward gradients become zero. This results in an irreversible loss of representational capacity, with up to 15% of network neurons becoming permanently dead. While alternatives like Leaky ReLU prevent absolute neuron death, they introduce negative noise accumulation and harsh gradient discontinuities at the origin that destabilize training.

The Solution: BAReLU

BAReLU introduces a Multi-Regime Fusion Framework that cleanly separates two behavioral domains:
  • Active Regime (x > 0): Pristine identity mapping identical to standard ReLU — zero feature distortion
  • Dormant Recovery Regime (x <= 0): Linear recovery lane with alpha = 0.01 — prevents neuron death while minimizing negative noise leakage
BAReLU's distinctive innovation is its Quadratic-Root System (sqrt(max(0,x)^2)) which conditions gradient flow at the origin boundary, producing smoother optimization dynamics than both ReLU and Leaky ReLU. The implementation uses branchless torch.where vectorization, achieving hardware execution parity with standard ReLU (less than 1.11% wall-clock overhead).

Key Results

CIFAR-10 (3 seeds: 42, 1337, 2026)
  • BAReLU: 68.32% accuracy, 0.00% neuron death | Standard ReLU: 66.47% accuracy, 15.03% neuron death
CIFAR-100 Fine-Grained (100 classes)
  • BAReLU: 31.39% accuracy, 0.00% neuron death | Standard ReLU: 29.57% accuracy, 12.11% neuron death
CartPole-v1 via PPO (389 policy updates)
  • BAReLU: 165.60 final reward | Standard ReLU: 138.60 | Leaky ReLU: 106.05
Gradient Conditioning Validation
  • -31.1% reduction in gradient norms vs Leaky ReLU
  • -38.3% reduction in weight update norms vs Leaky ReLU

Files

BAReLU_Academic_Paper.pdf

Files (875.1 kB)

Name Size Download all
md5:48cb68e3b75a0c5bb04e40ed560f773a
875.1 kB Preview Download