Isotropic Deep Learning: You Should Consider Your (Foundational) Biases

Presented is an alternative mathematical formulation, `\textit{Isotropic Deep Learning}', by analysing the geometric implications of current functional forms in deep learning. Contemporary networks rely near-universally on foundational forms respecting a discrete permutation symmetry. This is an underappreciated \textit{choice} in form, argued to introduce unrecognised biases. Initially, this discrete symmetry observation is promoted to a continuous rotation defined framework, then broadened to primitive sets defined by various other symmetries. This constitutes a new approach to symmetry for deep learning: Rather than enforcing data-derived symmetry transfer through model structure, it focuses on how the symmetries of foundational functional forms inherently \textit{act} on and \textit{interact} within, general architectures.

The distinct goal is to expose and leverage unintended inductive biases by deducing principles applicable in a broader, possibly universal context for beneficial computation. It proposes a systematic reformulation of \textit{all} foundational primitives to respect particular symmetries and understand their inherent effects, followed by a reselection of compositions upwards, potentially forming new models contingent on these alternative foundations. This constitutes a distinctly bottom-up reformulation aiming for broader, possibly universal, principles. This approach substantially contrasts with the established and successful group-theoretic paradigm, which constructs model-level symmetries tailored for niche, specialised applications requiring enforcement of end-to-end symmetry derived from a data structure. Thus, the objectives and resultant consequences differ between a bottom-up versus top-down consideration of symmetry's role. The philosophy is motivated by the prior demonstration that current functional forms influence activation distributions: discrete symmetries in functions induce a similar discrete structure in embedded representations through training. Thus, geometric artefacts can arise in learned representations solely due to human-imposed design choices rather than task-driven necessity. Therefore, the prevailing choice of form carries an unappreciated and unintended inductive bias. There appears to be no compelling a priori justification for why such representations or functional forms are universally desirable; this paper hypothesises three testable pathologies of the current formulation, with significant connections to mechanistic interpretability. These motivate constructing and analysing alternative foundational formalisms for function primitives, aiming to alter geometric constraints on representations and improve performance. The underlying inductive biases of a framework are suggested as a preferable default for adoption if a wide array of suitable and well-performing functions are developed. A variety of preliminary functions are proposed --- including new activation functions, normalisers, and operations. A symmetry-based audit of current primitives is also initiated. The symmetry-principled construction is then generalised, enabling a broad class of group-defined reformulations across sets of primitives, positing a new, axiomatic-like, foundational design axis with distinct inductive biases that underpin other design considerations. Thus, Isotropic Deep Learning becomes just \textit{one case study} among such parallel implementations for all models.

This initial group-theoretic generalisation of primitives is systematically extended upwards to encompass their hierarchical compositions, motivating consideration across all architectures. The framework provides three generations of symmetry strength for classification, categorising all functions and their compositions. Notably, this extension also recovers Geometric Deep Learning's approach as the strongest generation when considering compositions of functions to ensure model-scale compliance with the symmetry constraint. The formalism also recovers the Parameter Symmetry approach as a differing compositional consideration regarding consequences for parameters after a primitive algebra is fixed. It demonstrates both as distinct special cases characteristic of various compositional scales and strengths, thus unifying several contemporary approaches regarding symmetry in an intuitive, hierarchical, and complementary formalism.

This may facilitate comparison and exploration of their interplay, while clarifying further regimes that may remain to be considered. Presented as ``Taxonomic Deep Learning,'' this enables systematic audit and generation of distinct primitive classes from a broader, group-theoretic perspective. One can then systematically consider the emergent implications in the study of all downstream phenomena after a primitive algebra is fixed, including representation biases, theorems contingent on prior primitives, optimisation, performance, and diverse new model architectures.