Hi, this is Sourya Sen and I am here to present our project "Sounding Brush"
Existing applications of mobile music tools are often concerned with the simulation of acoustic or digital musical instruments
extended with graphical representations of keys, pads,
etc. In our tablet-based application "Sounding Brush,"
we implemented a digital drawing tool for music composition and performance. In our project
drawing is essentially the act of playing music and are inherently connected to each other
by means of a continuous process of observation,
individualization, and exploring time and space in a unique way.
Musical instruments in the field of electronic music are highly tied to their surrounding technological landscape.
In recent years,
this has led to the development of digital musical instruments, or DMI's, and research into new forms of interaction,
performance, and compositions. Through the employment of computers and portable computing devices,
different forms of interaction have been employed as control surfaces for musical instruments,
including the use of touch capable computing devices like graphic tablets, which are primarily aimed at illustrators and graphic designers.
However, a common feature among most of these interfaces is the continued use of digital
representations of physical controls and the use of linear timelines for time-based effects.
Direct and indirect relationships between visuals and music have been a matter of interest for many.
Outside the realm of moving images
commonly discussed under the umbrella term of visual music
Illustrations and graphic design for musical ideas have been explored by various composers
through graphic scores or graphic notations. In our application,
we combine these two fields towards real-time musical composition and performance application
which also functions as a drawing tool for exploring mark making.
This application
abstracts the natural process of drawing without becoming a controller but rather a platform for both musical expression and
time-based graphical notations,
with the visuals becoming a part of the performance itself.
Computationally powerful handheld devices have given a rise to the field of mobile music
Outside the two groups of mobile versions of desktop applications and emulation of real-world musical instruments,
there is a third group of instruments as well which take advantage of the unique capabilities
that a mobile device can offer.
Exploring relationships between drawing and sound characteristics have a long history. Most of the examples, however, are
approaches towards visualizing sound or synthesizing sound through graphical input and explore a one-to-one connection
in the time or frequency domain. Within the field of personal
computing the Kid Pix application is one of the earliest sonified drawing applications.
In recent years there have been various
Implementations of interactive music applications that use drawing or drawing-like gestures as an input method for musical applications.
Nevertheless the common themes that can be noticed are in the usage of either drawing as an input method in a simplified
Musical interface or treating sound as effects within primarily drawing applications.
Our application, "Sounding Brush," is implemented as an iOS application.
A few discrete components make up the software architecture, namely the synthesis engine,
a graphics engine and the communication layer in between them which update information in real time as the user is interacting with the application.
Presented as a white canvas to draw upon,
the user has access to different brushes which behave differently both sonically and visually,
Which the user can layer on top of each other,
akin to using different brushes and colors when painting on a real canvas.
The first brush is called "across," and it utilizes the simplest of audio-visual
mappings as well as synthesis techniques. Each instance of across is a sinusoidal wave
whose frequency is mapped with the horizontal coordinates of the tablet. While conceptually simple, this brush provides the user with
implementing additive synthesis techniques
The second brush is called "line," and it is very similar to "across" with one marked difference:
It is not a continuous waveform. As soon as the user finishes drawing a stroke the sound generation decays
"Three waves" utilizes subtractive synthesis the decay and filter envelope
Mapped to both the coordinates of the screen as well as color and character settings which mix different waveforms
Additionally the noise-based shader is utilized to provide a visual texture to the strokes that are drawn
creating a rich spectrum in both audio and visual domains
The fourth brush in the palette is called Kar+ Paint and it utilizes the Karplus-Strong synthesis technique to generate
String-like sounds. This is the only brush which generates a sound only once the user has finished drawing. This brush also
uses dynamic animated shaders with bi-directional mapping and the notes produced are quantized to a musical scale for playing pitched harmonies
"Particles" is based on granular synthesis and the visual strokes complement the sound by animating over time
Both within the mesh and the color both the aural and visual parameters evolve over time
giving the user certain unpredictability but is controllable to an extent that it is not purely random and
helps build texture and atmosphere within the performance
"Crackler" procedurally produces crackles, hisses, and drips based on procedural audio techniques
Visually the brush also morphs in and out of its original shape with animated textures and color gradients that evolve over time
Within the application, there are some features built in that both provide performance
opportunities as well as keeping memory utilization
in check. Each group of brushes have certain limits set on them like voices in a polyphonic synthesizer and
allow each group to be distinctly removed while keeping the others on canvas
Therefore it allows users to build up performances,
stacking up different instances of brushes and allows them to freely move between them to
dynamically change the audiovisual features over time.
Additionally the application also features the gesture brush which unlike the other brushes presented earlier
do not have any visual characteristics
but rather
utilize granular synthesis mapped to the movements of the tablet itself
Digital musical instruments and mobile music applications
primarily are targeted towards a certain group of people with a certain vocabulary and function within established norms of time and space.
For example,
most sound synthesis applications would use nomenclatures like frequency filter cutoff, etc.
Most digital instruments based on real instruments would provide graphical representations of keys, pads, etc. Our
Application breaks a lot of these ideas and strives towards a different understanding of performability,
first of all using drawing as the interface adds to the visual dimension and can appeal to both
musicians as well as visual artists. The act of drawing is the same as playing the instrument, not
separated from each other. Second, by combining both visual and sonic fields a performance can evolve over both time and space.
The temporal act of drawing, as well as the spatial positioning, feed into the process of sound generation
which in itself is a time-based process evolving through every successive instance of drawing.
Third, the barrier of entry is kept low: the act of drawing or scratching
the surface is an act that is a natural human activity and
approaching this instrument does not need any prior understanding of musical instruments or interfaces.
By utilizing both direct and indirect mapping strategies within each brush, learnability is also easily achieved.
Each successive brush builds on the mapping strategies used and, progressing through them, the user gains more
understanding and learns the application better. Finally,
there are performance strategies built into this application
to take advantage of its approach in
working within both time and space and allows the performer the opportunity to create a composition or build a performance
both in sound and visual domains
This application in its current state as presented here
Was both presented and demonstrated at the sound form symposium in Hamburg in March
2019. While there was a certain amount of interest that was expressed by the audience that
reinforced our ideas at the same time the project needs further development to be a comprehensive
musical instrument for a wide variety of audiences and performance. Towards that end, the next steps would include
comprehensive user testing from people with different backgrounds for evaluating the current state of the instrument and
understanding the future developments it would need to go through. Various other features
discussed during the development process but not implemented due to limitations of time would also be a focus
Some of these would be the ability to import samples for the granular engines
Saving and/or loading a composition, in-built recording and sharing functionality
Etc. This application was developed at Alto University with the support of the academy of finland and the sound form Instruments For All
Project. This project has also been released open source. Thank you