Journal article Open Access

Continual learning for recurrent neural networks: An empirical evaluation

Andrea Cossu; Antonio Carta; Vincenzo Lomonaco; Davide Bacciu


JSON-LD (schema.org) Export

{
  "inLanguage": {
    "alternateName": "eng", 
    "@type": "Language", 
    "name": "English"
  }, 
  "description": "<p>Learning continuously during all model lifetime is fundamental to deploy <a href=\"https://www.sciencedirect.com/topics/computer-science/machine-learning\">machine learning</a> solutions robust to drifts in the data distribution. Advances in Continual Learning (CL) with <a href=\"https://www.sciencedirect.com/topics/engineering/recurrent-neural-network\">recurrent neural networks</a> could pave the way to a large number of applications where incoming data is non stationary, like <a href=\"https://www.sciencedirect.com/topics/engineering/natural-language-processing\">natural language processing</a> and robotics. However, the existing body of work on the topic is still fragmented, with approaches which are application-specific and whose assessment is based on heterogeneous learning protocols and datasets. In this paper, we organize the literature on CL for sequential data processing by providing a categorization of the contributions and a review of the benchmarks. We propose two new benchmarks for CL with sequential data based on existing datasets, whose characteristics resemble real-world applications.</p>\n\n<p>We also provide a broad empirical evaluation of CL and Recurrent Neural Networks in class-incremental scenario, by testing their ability to mitigate forgetting with a number of different strategies which are not specific to sequential data processing. Our results highlight the key role played by the sequence length and the importance of a clear specification of the CL scenario.</p>", 
  "license": "https://creativecommons.org/licenses/by/4.0/legalcode", 
  "creator": [
    {
      "affiliation": "University of Pisa", 
      "@type": "Person", 
      "name": "Andrea Cossu"
    }, 
    {
      "affiliation": "University of Pisa", 
      "@type": "Person", 
      "name": "Antonio Carta"
    }, 
    {
      "affiliation": "University of Pisa", 
      "@type": "Person", 
      "name": "Vincenzo Lomonaco"
    }, 
    {
      "affiliation": "University of Pisa", 
      "@type": "Person", 
      "name": "Davide Bacciu"
    }
  ], 
  "headline": "Continual learning for recurrent neural networks: An empirical evaluation", 
  "image": "https://zenodo.org/static/img/logos/zenodo-gradient-round.svg", 
  "datePublished": "2021-08-05", 
  "url": "https://zenodo.org/record/5164245", 
  "keywords": [
    "continual learning; recurrent neural networks"
  ], 
  "@context": "https://schema.org/", 
  "identifier": "https://doi.org/10.1016/j.neunet.2021.07.021", 
  "@id": "https://doi.org/10.1016/j.neunet.2021.07.021", 
  "@type": "ScholarlyArticle", 
  "name": "Continual learning for recurrent neural networks: An empirical evaluation"
}
24
22
views
downloads
Views 24
Downloads 22
Data volume 34.9 MB
Unique views 22
Unique downloads 21

Share

Cite as