Learning music similarity using audio and textual input

The incredibly large amount of music that is available online nowadays has created a need for powerful retrieval and recommendation algorithms. Most music streaming services make use of matrix-factorisation collaborative filtering algorithms. While these systems are e˙ective, they have a number of limitations: the fact that they are content-agnostic leads to the cold-start problem (the model is unable to recommend new and unpopular songs), and the user cannot search for specific elements that can only be extracted from a song’s audio. The aim of this thesis was to develop a query-by-example system that can focus on specific aspects of a query song, such as ’instrumentation’, ’mood’, ’era’ or ’genre’, thus increasing user control. To this end, we compared two methods: (1) a feature-weighting method, that places higher weight on parts of song embeddings that are relevant to the aspect we want to focus on (2) a metric learning approach that encodes information about di˙erent aspects of a song in separate parts of its embedding. We applied the feature-weighting method to both content-based and collaborative filtering-based embeddings such as Weighted Matrix Factorisation (WMF) item factors, whereas we only the disen-tangled metric learning approach for content-based embeddings. The methods were evaluated quantitavely by inspecting tag co-occurrence between query- and retrieved songs, and qualitatively by inspecting retrieval examples and t-SNE visualisations of the song embeddings. It was found that WMF naturally groups together songs with shared semantic tags, and cannot be outperformed by the content-based dis-entangled metric learning method. However, the content-based method has the advantage of not su˙ering from the cold-start problem. Furthermore, the feature-weighting method was found to improve performance for both content-based tagging embeddings and WMF. Overall, we provide valuable insight into the di˙erences be-tween CF-based and content-based retrieval systems, and provide recommendations on how they can be combined efectively.