CLAP features for Audio Moment Retrieval

This page includes CLAP features of three datasets used in Language-based audio moment retrieval [1].

• Clotho-Moment
• UnAV100-subset
• TUT Sound Events 2017

Raw wav files are also publicly available here.

[1] H. Munakata, T. Nishimura, S. Nakada, T. Komatsu, "Language-based Audio Moment Retrieval", 2024, under review.

How to Use

We can train/evaluate audio moment retrieval models using these features in Lighthouse.
Please check the instructions of Lighthouse.

1. Unzip the file with the following commands
   Clotho-moment: 
   
   for file in clotho-moment_features.tar.part-*.gz; do gunzip "$file"; done

   clotho-moment_features.tar.part-* > clotho-moment_features.tar

   tar -xvf clotho-moment_features.tar
   
   UnAV100-subset, TUT Sound Events 2017: 
   
   tar -xvf tut2017_features.tar.gz

   tar -xvf unav100-subset_features.tar.gz
2. Set symbolic links in Lighthouse
   ln -s features/{dataset_name} {lighthouse_dir}/features

3. Train the model
   python training/train.py --model qd_detr --dataset clotho-moment --feature clap
   
   
4. Evaluate the model
   
   model=qd_detr
   

   dataset=unav100-subset

   feature=clap

   model_path={lighthouse_dir}/results/qd_detr/clotho-moment/clap/best.ckpt

   eval_split_name=val

   eval_path=data/unav100-subset/unav100-subset_test_release.jsonl

   
   python training/evaluate.py \

   --model $model \

   --dataset $dataset \

   --feature $feature \

   --model_path $model_path \

   --eval_split_name $eval_split_name \

   --eval_path $eval_path