A polyphonic pitch tracking embedded system for rapid instrument augmentation

This paper presents a system for easily augmenting polyphonic pitched instruments. The entire system is designed to run on a low-cost embedded computer, suitable for live performance and easy to customise for different use cases. The core of the system implements real-time spectrum factorisation, decomposing polyphonic audio input signals into music note activations. New instruments can be easily added to the system with the help of custom spectral template dictionaries. Instrument augmentation is achieved by replacing or mixing the instrument's original sounds with a large variety of synthetic or sampled sounds, which follow the polyphonic pitch activations.