Orthogonality in Additive Echo State Networks

Reservoir computing (RC) is a state-of-the-art approach for efficient training in temporal domains.
In this paper, we explore new RC architectures that generalise the popular leaky echo state network model (leaky-ESN) introducing an additive orthogonal term outside the nonlinear part of the ESN equation. We investigate the benefits of employing orthogonal matrices in ESNs both inside the nonlinearity and outside of it. We show empirically how to boost the memory capacity towards the theoretical maximum value while still preserving the power of nonlinear computations. Ergo, we optimise the compromise between computing with memory and computing with nonlinearity.
The proposed model demonstrates to outperform both leaky-ESN and orthogonal reservoir ESN models on tasks requiring nonlinear computations with memory.