10.5281/zenodo.5723053
https://zenodo.org/records/5723053
oai:zenodo.org:5723053
Jan Berg
Jan Berg
Audio Research Group, Tampere University, Finland
Konstantinos Drossos
Konstantinos Drossos
Audio Research Group, Tampere University, Finland
Continual Learning for Automated Audio Captioning Using The Learning Without Forgetting Approach
Zenodo
2021
automated audio captioning
continual learning
learning without forgetting
WaveTransformer
Clotho
AudioCaps
2021-11-15
eng
https://github.com/JanBerg1/AAC-LwF
10.5281/zenodo.4783391
10.5281/zenodo.5770113
10.5281/zenodo.5723052
https://zenodo.org/communities/marvel_project
https://zenodo.org/communities/eu
Creative Commons Attribution 4.0 International
Automated audio captioning (AAC) is the task of automatically creating textual descriptions (i.e. captions) for the contents of a general audio signal. Most AAC methods are using existing datasets to optimize and/or evaluate upon. Given the limited information held by the AAC datasets, it is very likely that AAC methods learn only the information contained in the utilized datasets. In this paper we present a first approach for continuously adapting an AAC method to new information, using a continual learning method. In our scenario, a pre-optimized AAC method is used for some unseen general audio signals and can update its parameters in order to adapt to the new information, given a new reference caption. We evaluate our method using a freely available, pre-optimized AAC method and two freely available AAC datasets. We compare our proposed method with three scenarios, two of training on one of the datasets and evaluating on the other and a third of training on one dataset and fine-tuning on the other. Obtained results show that our method achieves a good balance between distilling new knowledge and not forgetting the previous one.
The authors wish to acknowledge CSC-IT Center for Science, Finland, for computational resources. K. Drossos has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 957337, project MARVEL.
European Commission
10.13039/501100000780
957337
Multimodal Extreme Scale Data Analytics for Smart Cities Environments