This project is entitled Cyclops: Designing an Eye-Controlled Instrument for Accessibility
and Flexible Use.
My name is Willie Payne.
I am a PhD Candidate in Music Technology at New York University.
My collaborators in this project are Ann Paradiso, Principal User Experience Manager in the Microsoft
Research Next Enable Team - and Shaun Kane, Associate Professor of Computer Science at
the University of Colorado Boulder.
I am making this presentation from my studio apartment in late May and early June while
much of the city, including NYU, remains on lockdown due to COVID-19.
Please forgive the limited audio quality as I do not have access to my usual equipment
and facilities.
The Cyclops is an eye-gaze controlled instrument designed for live performance and improvisation.
It is primarily motivated by a need for expressive musical instruments that are more easily accessible
to people who rely on eye trackers for computer access, such as those with late-stage amyotrophic
lateral sclerosis or ALS.
In this presentation, I will touch on the background, development, and design thinking
of the Cyclops, provide a short demo overview, and conclude with suggestions for those hoping
to develop eye-controlled musical interfaces.
Our partner and muse throughout this work, and the Hands-Free Music Project at Microsoft
Research as a whole, is Jeremy Best, a musician who plays in the band Dangerfield Newby.
Jeremy was diagnosed with ALS about six and a half years ago, and at this late stage is
unable to move virtually all of his muscles relying on a ventilator to breathe.
Designed for Jeremy, and those like Jeremy, the Cyclops is an eyes-first interface meaning
it functions with eye-gaze alone, and may be extended to support other inputs for users
with more mobility.
Eye-gaze interfaces present a number of interaction challenges that are amplified in musical applications.
The input from an eye-tracker is an x,y point collected at a sampling rate with varying
levels of accuracy resulting from multiple factors.
Commercial eye trackers use smoothing algorithms and can be calibrated to improve accuracy,
but the onus to make an interface usable still falls heavily on the designer.
In the Cyclops, targets are made especially large and placed near the center of the screen
where research has shown trackers tend to be more accurate.
Furthermore, without a switch, activating an element is usually done by dwelling over
it for an extended period of time.
Longer dwell times improve accuracy preventing a phenomenon called Midas Touch, where the
user accidentally activates unintended elements in their path along the screen, but they introduce
undesirable lag preventing the user from triggering musical events immediately.
Dwell/timing tradeoffs can be mitigated through considered element placement and dwell times
may dynamically respond to user ability and preferences.
Finally, designers must consider the actual human needs of their users.
Eye gaze can be heavily fatiguing, exacerbated by certain interactions like rapid motion
"eye-drumming" or unpleasant animations or colors.
The Cyclops mitigates risk of fatigue through multiple modes of play that vary in degrees
of effort.
To learn more about designing eye-controlled music interactions, we encourage interested
viewers to read our paper and to look to Anthony Hornof's paper entitled, The Prospects For
Eye-Controlled Musical Performance, from the 2014 NIME proceedings for a more comprehensive
overview of prior developments, challenges, and strategies.
Many, if not most, who rely on eye trackers use Windows due to the operating system's
built-in support for eye control.
The Cyclops is developed using Universal Windows Platform giving it compatibility with existing
tracker-enabled devices and commercial eye trackers like the Tobii Eye Tracker 4c and
Tobii PCEye Mini.
Furthermore, UWP supports the open source Gaze Interaction Library which makes the process
of creating and fine tuning the behavior of gaze-enabled elements simple yet precise.
Unfortunately, UWP only possesses a very limited audio engine in its Audiograph API.
For audio synthesis and sample accurate timing, the Cyclops embeds a hidden, offline web browser
that generates audio through the Web Audio API.
The Cyclops began with two primary design goals: First and foremost, it should possess
a low barrier of entry, and should be as easy and accessible as possible.
Second, it should still provide a degree of expressive control with room for growth as
the musician plays with it longer.
While the paper goes into much more detail, I will describe two examples for how our design
attempts to achieve these goals in its layout and note input.
First, the Cyclops separates control into three screens: an instrument screen for note
input, a sequencer screen, and an effects screen which is made of a two-dimensional
canvas resembling a Korg Kaoss Kontrol pad.
Early in development, we were heavily influenced by the Dato DUO, a synth designed for two
players who sit across from each other and handle mutually exclusive sets of controls.
The Cyclops similarly divides control across multiple screens, allowing targets to be rendered
extra large and sectioning off different kinds of musical interaction.
In reducing a two-player interaction to a single-player one , we allow certain parameters
to be automated and looped.
Every note inputted is automatically added to the sequence, while gaze paths on the effects
screen are recorded and can be looped.
Aware of notable gaze interaction challenges like Midas touch, we experimented with a number
of layouts, dwell times, and types of controls on the instrument screen.
We ultimately opted for a two-dimensional grid layout inspired by string instrument
fretboards for its understandability and flexibility.
The example shown on screen is a diatonic scale.
Horizontal gaze motion results in playing the notes of the scale while vertical motion
results in arpeggios.
Diagonal motion affords other intervals.
Furthermore, the grid itself toggles between three different dwell settings.
The slowest setting is preferable when learning a new tune or when precision is important,
while the fastest setting supports improvisation and when the user just wants to play a bunch
of notes.
Future iterations of the Cyclops may provide even more user customization over dwell times,
the number of columns, supported scales, and relationship between rows.
I will now show a short demo video of the Cyclops that is also linked in the paper.
The Cyclops is an eye gaze-controlled synthesizer and sequencer intended to support live play
and improvisation.
The Instrument screen is inspired by string instrument fret boards.
Horizontal movement produces steps while vertical movement produces arpeggios making note input
easy and intuitive.
Oh, that little pink dot is where the eye tracker thinks I'm looking.
Many parts of the Cyclops are loopable, and in fact that melody I played was recorded
and can be played back.
Using the sequencer screen, we can fine tune our loop changing the tempo or adding accents
or rests.
The gate sequencer is independent from the pitch sequencer, and each has an adjustable
length.
Certain number combinations can result in long, complex sequences.
Finally, the effects screen allows two-dimensional effects control.
My actions are again automatically recorded and can be played back
on loop.
With these building blocks, the Cyclops is capable of quite a wide range of sounds.
We were able to informally test the Cyclops with Jeremy in February, and can report positive
results helping justify future developments of the Cyclops.
Working with people with late-stage ALS requires certain considerations.
In Ann's experience, Jeremy, and others, often do not allow prototypes to be deployed on
their personal devices.
Their device is their primary means if not only means of interacting with the world around
them.
It is understandable that they do not want developers to change settings around or inadvertently
mess things up.
Many people who Ann has worked with have also not been willing to upgrade their devices.
Thus, to conduct user testing, developers must prepare their own hardware, ask questions
well in advance to ensure the participant is comfortable during their visit, and schedule
sessions when trusted caretakers and companions can also attend.
During a session, developers should recognize communication with eye typing can be slow
and frustrating, limiting user feedback at least right away, and prepare multiple kinds
of activities such that a high degree of attention and eye tracker ability is not necessary to
lead to any valuable findings.
In this presentation, I discussed the Cyclops, an expressive eye-controlled synthesizer intended
to enable live improvisation and play for those who rely on eye trackers for computer
access.
Thank you so much for your time and attention.
Please do not hesitate to reach out with any feedback or questions.