Applying Computer Vision Foundation Models to Audio Source Separation

In this paper we attempt to apply foundational models in computer vision to the problem of audio source separation. Focusing on separating a foreground sound from a mixture, we attempt to leverage the strong feature learning capabilities of these models to perform image segmentation on audio spectrograms. Using an augmented version of the FSD50k dataset, we first asses the new models DINOv2 and the SAM
(Segment Anything Model) in a zero-shot transfer learning scenario, then under different fine-tuning methods. We hope to find state-of-the-art performance.