AI in live coding environments: Pandora's Dream

https://aimc2023.pubpub.org/pub/9lqurr1x

Pandora hears her dreams, they talk to her in mysterious voices and unknown languages. You are there in the middle of the dark but you don't know how did you get there, are you one of Pandora's dreams? Talk to her, maybe she will answer you...

Pandora's Dream is a versatile live coding playground that opens up a world of possibilities for performers seeking to engage audiences through live coding performances. Designed as a dynamic and interactive platform, Pandora empowers artists to harness the perspective of multiple tools, languages and approaches to create audiovisual experiences in real time.

Built within the Unity engine, Pandora serves as a text editor that transcends conventional coding practices. Its design enables performers to seamlessly weave together various programming languages, including C#, Javascript, GLSL, ChucK, and even a subset of C# called Pandora. This diverse array of languages empowers artists to explore different creative avenues and leverage the unique features and capabilities offered by each language.

Pandora also has an API to create other APIs to communicate and interact with other platforms and tools, and due to this, it was possible to develop an API to handle AI/machine learning algorithms in live coding environments. This is the case for CHAI, part of ChucK language released in version 1.5.x.x.

The integration with CHAI was tested by training an unsupervised model that can be fed in real-time with a live coding improvisation, using the audio output. In this case, the model was trained on a collection of 1000 audio samples mainly part of the Dirt collection (https://github.com/tidalcycles/Dirt-Samples) plus some samples collected by the author. The training time was no more than 10 seconds extracting classical audio features including Centroid, RMS, and Mel Frequency Cepstral Coefficients. Then using Similarity retrieval the algorithm creates an audio layer that fits the improvisation in real time.