Andrea Cossu
Antonio Carta
Vincenzo Lomonaco
Davide Bacciu
2021-08-05
<p>Learning continuously during all model lifetime is fundamental to deploy <a href="https://www.sciencedirect.com/topics/computer-science/machine-learning">machine learning</a> solutions robust to drifts in the data distribution. Advances in Continual Learning (CL) with <a href="https://www.sciencedirect.com/topics/engineering/recurrent-neural-network">recurrent neural networks</a> could pave the way to a large number of applications where incoming data is non stationary, like <a href="https://www.sciencedirect.com/topics/engineering/natural-language-processing">natural language processing</a> and robotics. However, the existing body of work on the topic is still fragmented, with approaches which are application-specific and whose assessment is based on heterogeneous learning protocols and datasets. In this paper, we organize the literature on CL for sequential data processing by providing a categorization of the contributions and a review of the benchmarks. We propose two new benchmarks for CL with sequential data based on existing datasets, whose characteristics resemble real-world applications.</p>
<p>We also provide a broad empirical evaluation of CL and Recurrent Neural Networks in class-incremental scenario, by testing their ability to mitigate forgetting with a number of different strategies which are not specific to sequential data processing. Our results highlight the key role played by the sequence length and the importance of a clear specification of the CL scenario.</p>
https://doi.org/10.1016/j.neunet.2021.07.021
oai:zenodo.org:5164245
eng
Zenodo
https://zenodo.org/communities/teaching-h2020
https://zenodo.org/communities/eu
info:eu-repo/semantics/openAccess
Creative Commons Attribution 4.0 International
https://creativecommons.org/licenses/by/4.0/legalcode
Neural Networks, 143, (2021-08-05)
continual learning; recurrent neural networks
Continual learning for recurrent neural networks: An empirical evaluation
info:eu-repo/semantics/article