Spectral Ergodicity in Deep Learning Architectures via Surrogate Random Matrices

The new methodology combines approaches inspired from the Thirumalai-Mountain (TM) metric and
metric rooted in Kullbach-Leibler (KL) divergence. The method is applied to a general study of
deep and recurrent neural networks via the analysis of random matrix ensembles mimicking typical
weight matrices of those systems. In particular, we examine different size circular random matrix
ensembles: circular unitary ensemble (CUE), circular orthogonal ensemble (COE), and circular
symplectic ensemble (CSE). Eigenvalue spectra and spectral ergodicity are computed for those
ensembles as a function of connectivity in the system.  The dataset and Python notebook are provided.