Published January 15, 2022 | Version v1
Journal article Open

Architectural richness in deep reservoir computing

  • 1. University of Pisa

Description

Reservoir computing (RC) is a popular class of recurrent neural networks (RNNs) with untrained dynamics. Recently, advancements on deep RC architectures have shown a great impact in time-series applications, showing a convenient trade-off between predictive performance and required training complexity. In this paper, we go more in depth into the analysis of untrained RNNs by studying the quality of recurrent dynamics developed by the layers of deep RC neural networks. We do so by assessing the richness of the neural representations in the different levels of the architecture, using measures originating from the fields of dynamical systems, numerical analysis and information theory. Our experiments, on both synthetic and real-world datasets, show that depth—as an architectural factor of RNNs design—has a natural effect on the quality of RNN dynamics (even without learning of the internal connections). The interplay between depth and the values of RC scaling hyper-parameters, especially the scaling of inter-layer connections, is crucial to design rich untrained recurrent neural systems.

Files

s00521-021-06760-7.pdf

Files (3.3 MB)

Name Size Download all
md5:5fae6020e864b484be2c14f46aa83899
3.3 MB Preview Download

Additional details

Funding

TEACHING – A computing toolkit for building efficient autonomous applications leveraging humanistic intelligence 871385
European Commission