URack is a modular, audio-visual composition and performance environment that combines
the Unity video-game engine with the VCV Rack software modular synthesiser.
This project allows sound designers and musicians familiar with modular synthesis to explore
the creation of real-time visuals without needing to learn graphics programming, or
utilise a separate software environment for visual composition.
This simple patch here shows how a drum sequencer is able to drive percussion modules, as well
as URack modules responsible for generating 3D graphics.
This demonstrates how easy it is to integrate real-time visuals in existing composition
and performance workflows.
Modules usually only responsible for sound-sculpting can be plugged directly into graphics.
The Unity game engine allows deployment to a wide range of platforms, including PCs,
consoles, mobile, and XR devices.
Here, the very same patch is being streamed to a Microsoft HoloLens headset to provide
a hands-free augmented reality experience.
URack abstracts graphical composition behind a custom VCV Rack plugin.
URack modules live alongside thousands of other modules available in VCV Rack dedicated
to sound and control signal creation.
Communication between VCV Rack and Unity works using Open Sound Control (OSC) over a UDP
network connection.
When OSC messages are received at the Unity instance, any publicly exposed script property
or VFX-Graph property addressed will be updated on the next frame.
URack supports the connection to an arbitrary number of Unity-based clients, allowing for
simultaneous presentation on multiple devices, as well as multi-user, shared experiences.
URack extends the VCV Rack API by introducing point-clouds as first-class value types.
On top of audio and control signals, URack modules have the ability to pass point-cloud
streams between each other.
Point-clouds can be sourced from pre-recorded sequences, or in real-time using commodity
RGBD sensors such as the Microsoft Kinect, or Intel Realsense cameras.
In this example, a real-time point-cloud of the performance environment is visualised
using URack modules, and gestural controllers are used to manipulate the point-cloud visualisation,
and to navigate through the scene.
VCV Rack contains many modules that allow interoperability with external sound devices.
The audio from this analog bass synthesiser is routed through a filter bank to track its
brightness, which is used control the appearance of a prerecorded point-cloud.
More modules are added to increase the complexity of this scene.
The
waves on the floor and ceiling here are being animated by the MIDI output of this hardware
drum-machine.
This demonstration shows how URack can be used to transform existing hardware performance
setups into mixed-reality, audio-visual instruments.
URack includes tools and an API for developers that makes it easy to start building modules.
A python script is provided that converts panel designs in the form of vector graphics
into functioning URack modules, without needing to write VCV Rack C++ or networking code between
the host and client.
The documentation and source code for URack is available online, and we are currently
looking for testers and contributors.
Thanks for watching, and thanks to NIME 2020.