The Thummer Mapping Project (ThuMP)

This paper presents the Thummer Mapping Project (ThuMP), an industry partnership project between ThumMotion P/L and The University of Western Sydney (UWS). ThuMP sought to developing mapping strategies for new interfaces for musical expression (NIME), specifically the Thummer#8482;, which provides thirteen simultaneous degrees of freedom. This research presents a new approach to the mapping problem resulting from a primary design research phase and a prototype testing and evaluation phase. In order to establish an underlying design approach for the Thummer#8482; mapping strategies, a number of interviews were carried out with high-level acoustic instrumental performers, the majority of whom play with the Sydney Symphony Orchestra, Sydney, Australia. Mapping strategies were developed from analysis of these interviews and then evaluated in trial usability testing.


INTRODUCTION
The ThuMP project is engaging in the development of a new electronic musical interface/instrument based on a re-evaluation of the performer's relationship with the performance interface. It sought to go back to examine musical interfaces that are broadly agreed to be successful and have persisted for a long time; acoustic instruments, namely, string and wind instruments and because of the nature of the Thummer™ interface, the piano accordion and concertina.
The ThuMP project posits that approaching the challenge of musical interface design from the musician's perspective might enable a detailed understanding of the subtle mechanisms of feedback and control that allow and support virtuosic technique.
There exists a need to combine the valuable research outcomes from the computer sciences community and the musician's perspective at a semantic level -one approach to this problem is based on temporal data, such as the measuring of sensor pressure and angle over time, rates of velocity, acceleration and other quantifiable, measurable characteristics, however, the data itself is already conditioned by interface design decisions. In order to address these inbuilt biases, the ThuMP project has asked several questions of highly skilled acoustic musicians and analysed the interview data using qualitative software tools.
Approaches to evaluating the resultant mappings have been explored in a pilot playability study.

THE RESEARCH QUESTIONS
The research was conducted in two stages. The first stage identified, quantified and categorised the perceived control gestures [1,5,7,[16][17][18][19][20][21] for a number of classes of musical instruments (wind, string and piano accordion/concertina) from the performer's perspective. The second stage employed the stage one data to develop experimental mapping strategies for a single voiced instrument controlled by the Thummer™ interface. These mappings [2,8,9,14] were then tested in an attempt to develop a playability model to underpin further evaluation.

Participants
Semi-structured interviews were carried out with professional musicians during June 2005 -July 2006. A total of nine (n = 4 male; n = 5 female) tertiary trained musicians participated in the initial stage of the research. All participants were involved in the teaching of their instruments, in addition to performing their instrument professionally. The length of time spent playing the instrument ranged from seven years to 30+ years. Participants were experts in the field of flute (n = 2 female); double bass (n = 4 male); violin (n = 2 female); and piano accordion (n = 1 female), and ages ranged from under 25 years to 55+ years.

Materials
A semi-structured interview schedule was devised by the first author (in consultation) and covered the following broad questions: 1.
What instrument do you play? 2.
An important aspect of learning to play your instrument is to develop control over the sound of the instrument. What aspects of the sound of your instrument are controllable? 3.
When you are practicing your instrument and developing your technique, what are the physical controls that you exercise in manipulating the instruments' controllable sound properties? Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. NIME07, June 7-9, 2007, New York, NY Copyright remains with the author(s).

4.
When you are playing your instrument in a performance, what physical controls do you consciously exercise in the manipulation of the controllable sound properties?

5.
To what degree are these physical controls independent or inter-related? These questions were designed to interrogate the following principal characteristics of instrumental mappings:

Stage One
1) How many discrete control parameters do trained musicians and teachers consciously exercise in normal performance conditions? This question begins to define existing models of musical gesture space, identifying direct control, levels of emergence and possible 'uncontrol'; and 2). How do these parameters directly relate to audible timbral characteristics? This question re-assesses existing models of timbre space from the performer's perspective [19]

Procedure
Participants were recruited through professional music organizations. Those that were interested were informed about the study, its goals and the interview process. Each participant was issued with an informed consent form, a participant information sheet and a general demographic questionnaire. Interview times ranged from 35-110 minutes in length. The interviews were conducted at a time and location suitable to each participant, and took place in a quiet area free from distraction. At the conclusion of each interview participants were supplied a reply paid envelope and additional paper in the event that they would like to add to, or clarify, any of the issues raised during the interview.

Stage One Results
The transcripts of these interviews were analysed for musical concepts using the Leximancer [15] and qualitative analysis software solutions suited to emergent methodologies for doing discourse analysis, grounded theory, action research, conversation analysis, ethnography, phenomenology and mixed methods research.
A basic Leximancer analysis of the initial interviews (September 2005) identified a list of shared terms used by musicians to define the principal controlled parameters of the target acoustic instruments as: Tone, (tone colour, sound colour (resonance), tone quality), dynamics, volume, expression, duration, vibrato, articulation, attack, release, sustain, pitch and intonation.
The concepts are based on a lexicon of seed words entered by the researcher(s) and the software's concept learning routine, which discovers relevant text segments that do not contain the actual seed terms identified by the user, providing automatic taxonomy discovery and concept cluster mapping by applying Bayesian theory to the interview transcripts.
Automated concept mapping was undertaken to discover primary control mechanisms on successful acoustic instruments without bias from the first author, who has conservatorium training. The Leximancer concept map can be adjusted to show differing levels of concordance, subsequently varying the number of concepts displayed. In this way the map in Figure 1 can be reduced to that shown in Figure 2, which is useful in determining the primary concepts and developing hypotheses from the interview data.
It can be seen in Figure 1 that concept clustering places the two flute players on one side of the concept map and the violin and double bass on the other. Equally concepts common to these instruments; air for the flute and bow for the strings, are weighted towards the appropriate instruments whilst shared concepts such as timbre, sound, pitch, control, tone, pressure and colour indicate concordances for all interviewees. The concept maps are being continually refined in an effort to gain greater clarity regarding the relationships between control parameters and audible timbral characteristics, however it can be seen that relationships have been established using asymmetric concordance and that these concepts appear to remain musically useful.
This approach facilitates an exploration of the playability of high-dimensional control spaces by mapping control dimensions to cognitively meaningful timbral variables. The 'Playability' studies fed iteratively back into the mapping strategies, providing a performer perspective, determining the feasibility and value of control mappings on the basis of musical literacy.

Reviewing Stage One Data Analysis
The results produced by the Leximancer software were so close to those expected by the first author that it raised concerns that the hypothesis was too heavily influenced by the author's conservatorium training and did not reliably represent the actuality of the interview data.
The scientific model posits a hypothesis/test/analyse model, however as the first author had: written the interview questions; conducted the interviews; directed the data analysis and in the light that no control group existed, it was decided to undertake a second analysis phase with a highly experienced qualitative data analyst (a non-musician, the third author) and the NVivo [13] tool.
This process depended much more heavily on human perception and experience being brought to bear as the principal pattern recognition tool. The third author marked up all text to define the focus of each interview and then utilised the software to refine and quantify those interpretations as a secondary measure.
The second analysis phase produced detailed diagrams of control parameter relationships to tone colour and acknowledges the importance of musicianship as is outlined below (see Fig.3).

Analysis -An alternative approach
The second approach to interview analysis used the NVivo software tool. Content analysis was conducted on the data in a series of phases.

Phase 1 -Individual qualitative analysis
Phase 1 involved each interview being analyzed individually by a researcher who was skilled in qualitative data analysis, yet a layperson regarding musicianship. This was done to reduce bias in the analysis. During this phase the researcher thoroughly read each transcript an average of five times to immerse themselves in the data. The parameters of control that the participant exercised over their instrument were noted. These included; pitch, dynamics, articulation (attack, release, sustain) and, vibrato. Through further content analysis, based solely on the logic outlined in each discourse, the physical controls that each musician utilized to affect these control parameters were also noted. In addition, the interconnections between these controls, and the overall effect on the sound of the instrument were distinguished. This part of the analysis was then represented diagrammatically, noting the above connections and inter-relatedness of the physical controls, the control parameters and their effect on the overall sound of the instrument (see for example figure 3) Using the NVivo qualitative data analysis program, each of the pathways outlined diagrammatically was then supported with transcript data. For example, figure 3 indicates that pathway 6 concerns the embouchure and its affect on the dynamics of the flute. As stated by the participant: For softer, smaller embouchure. Less air than is required for the loud playing but still enough air so that the note works. Also, the angle of the air generally angles upwards.
The transcribed discourse was then subject to a summary analysis, so that each pathway was succinctly represented. For example, pathway 6 was summarized to include: A smaller embouchure is used to play softly -because the air still has to come out fast. There is less air than when playing loud, but still enough to make the note work. The air is angled upwards.
To play loudly, more air is required, that is, the embouchure gets larger to allow more air and the air is angled downwards.

Figure 4 Common underlying physicality involved in controlling sound dynamics
At the conclusion of the analysis outlined above, each individual interview data set and subsequent pictorial representation was discussed and cross-checked with the primary researcher (Dr Paine) and one other researcher (Stevenson) who were both experienced in music research and had knowledge in the area. This served to validate the representations and qualitative analysis, providing some degree of inter-rater reliability.

Phase 2 -commonalities
A second round of interviews was then conducted with the instrumentalists from phase one in order to clarify the relationships between the physical controls of the instrument, the defined principal control parameters (dynamics, pitch, vibrato, articulation, release, attack) and the tone colour as outlined in Figure 4. The questions were: 1.
Which instrument do you play? 2.
An important aspect of learning to play your instrument is to develop control over the sound that your instrument produces. If these are the controllable aspects of your instrument (show participants the list) and you may like to add or delete some of these, what I'm interested in is what are the physical controls that you exercise in the manipulation of these properties. So we can start with dynamics…. [Proceed through list]

3.
If you had to rank these factors from most important to lest important, how would you rank them? So is dynamic more or les important than pitch?

4.
Do you think these factors (dynamics, pitch etc) are independent or interrelated? If they interrelated, how are the connected?

5.
How do each of these controllable properties (dynamics, pitch) influence or affect the overall tone colour? And again, perhaps we can go through them one by one, so starting with dynamics.
These questions sought to clarify: 1) What are the useable number and range of control parameters offered by the prototype Thummer™? This question defines the playable range or gesture space of the prototype Thummer™?
2) What influence do control-mapping strategies [2,8,9,14] have on the players ability to complete a simple performance gesture task?
In phase 2 of the analysis the interviews were analyzed simultaneously. The primary aim of this phase was to identify the commonalities amongst the interviews in regards to controllable sound properties, and the physical controls that are exercised in the manipulation of these properties. With regards to controllable sound properties, four parameters were consistently noted across all the interviews, hence providing a robust result. These were: dynamics, pitch, vibrato and articulation (including attack, release, sustain). (see figure 4).
With regards to the physical controls that are exercised in the manipulation of these properties, a number of commonalities were identified. However, given the variance evident in the physical manipulation of the instruments included in the study (for example, the flute and the double bass), the commonalities identified were based on similarities in the underlying physicality of the process involved. To illustrate; in controlling the sound dynamics, double bass players vary the amount of bow hair used to impact the string by varying the angle of the bow (relative to the string); Flute players vary the angle of the air stream. The underlying physical process across these two manipulations can then be identified as variance of angle. This type of analysis was repeated for each of the four control parameters outlined above, and was again represented diagrammatically (see for example, Figure 3).
In phase 3 of the analysis, the commonalities identified amongst the interviews, for each of the controllable sound properties, were then scrutinized for higher order or general explanatory controls that could be logically mapped onto the Thummer. These included: angle, pressure, speed and position, and were used to control the dynamics, pitch and vibrato, and articulation (including attack and release).
In summary, figure 4 represents a generalised model of the control parameters identified in the interviews using the NVivo approach, all of which depend on the pragmatics of the instrument in questions (i.e. bowing technique and air stream control) but which determine the most critical musical attribute, the overall tone colour. The principal controls being:

Dynamics, Pitch, Vibrato, Articulation, and Attack and Release
These differ from the previous Leximancer analysis only slightly, with Tone Colour being seen here not simply as a variable, but the principal objective of all control, and Dynamics and Volume, Expression, Duration and Intonation falling under more general concepts such as Pitch, Dynamics and Articulation.
Inter-relationships exist within even the most generalised model, and in asking musicians to identify the inter-relationships of the myriad specialist control parameters relating to their instrument, they often commented that they were all inter-related -that very little could be done by isolating a single parameter.

Design Recommendations
The common physical instrument controls identified in this study are (Figure 3, figure 4 and Figure 6):

Pressure, Speed, Angle and Position
As briefly mentioned above, these are applied in different ways for each instrument, however an increase in amplitude on a string and wind instrument requires a change in angle and pressure of the excitation force. The bow is angled to place more hair on the string and more pressure is applied, or the air stream is directed more into the instrument and the pressure is increased so that more air travels through the instrument.
These parameters not only represent the key variables in controlling the timbre of the instrument (i.e. Dynamics, Pitch, Vibrato, Articulation (inc. Attack and Release), but represent key cognitive affordances [6], associated with playability and control mapping; affordances that have developed over several centuries in instruments that have persevered and provide discernable, just-noticeable-difference control over timbre, both individually and in combination.
In the light of these analyses, it is recommended that Thumtronics examine the way in which Pressure, Speed, Angle and Position could be provided as first order control parameters on the Thummer.
These characteristic are most easily equated to the following: • Pressure can be applied to the button field as aftertouch or key velocity, but it could also be applied to the sides of the Thummer instrument so that the performer could control timbre though squeezing the instrument with the palms of the hands.
• Speed can again be equated to key velocity, but would also be usefully associated with the movement of the Thummer instrument through a three-dimensional plane in front of the performer. The 3DS [11] sensor provides the control input and it is highly recommend that Thumtronics look for economic avenues to make this variable available. The control parameters could include both absolute speed of movement and acceleration. Acceleration would be usefully applied to the attach or release envelop of as note, whilst velocity could be used for portemento/glissando, trills, vibrato and tremolo in note based music or used to control the playback speed of samples, the ramp or envelop or filter rate or degree of applied modulation.

•
Angle is clearly related to small variations in pitch; the flattening or raising of the pitch in quarter or semitone glissandi. It may also reasonably be applied to modulation rates to vary the timbre or control vibrato, however the application of angle to overall amplitude would also allow for small nuancing during a phrase independent of articulation and timbral characteristics.

•
Position can be applied here to finger position in the button field controlling pitch or joystick position.
The application of position is always in relation to one of the three characteristics listed above, and could be applied to the Thummer in terms of tipping the instrument inwards towards the body or outwards, however this is quite difficult to do in practice, and may be better applied via the thumb-sticks on the back of the Thummer.
The current prototype of the Thummer instrument does not allow for these characteristics to be applied in a direct manner, with the exception of key velocity and after-touch. The ability to apply Pressure, Speed and Angle as indicated above would assist in differentiating the Thummer from all other interfaces on the market.
It would also provide for the foundation characteristics identified in this study. It is felt that these characteristics should be implemented through an immediate, first-order mechanism and not through an abstraction such as speed = thumb-stick 1, angle = thumb-stick 2 etc The abstraction of tangible control parameters is a common shortfall of experimental electronic musical interfaces. The analysis of acoustic instruments in stage one of this project show that successful instruments have a direct, discernable relationship between the excitation moment and the difference that gesture has on the sound. This study has differentiated a number of principal components of control that we believe are critical to all successful instruments and should therefore form the cornerstone of the Thummer's design ideology.

Background
The second stage addressed practical aspects of the Thummer™ instrument itself. During the project two prototypes were received. Changes in the design of the instrument reflected responses to early user testing, developments in the concept of the instrument, and refinements influenced by production considerations. Notable changes included a dramatic weight reduction in the later prototype to better support handheld performance; the development of single piece pressure sensitive conductive plastic key field technology (in response to both user feedback on the tactile properties of the earlier button/spring design and cost imperatives for mass manufacture); and the most significant change being the division of the instrument into two halves each housing a complete key field, joystick and associated buttons. This change dramatically expanded the potential application of the instrument in performance. The manufacturer intends to allow the user to define the way in which the instrument is used. Possible approaches include table or stand top use with the key fields placed side-by-side, handheld in the manner of an accordion (but without the bellows) or using a neck strap and mounting support so the two hands can be used in the manner of a guitar. While this open approach to the development of the instrument is to be applauded and may be a contributing factor to the potential commercial success of the instrument, it presented some challenges in the design of this research. Rather than assessing all possibilities we chose to adopt a fixed arrangement with the two halves joined back to back.
The basic unit was augmented by a commercial gyro enhanced orientation sensor [11].
Thumtronics have put significant effort into assessing and designing solutions to problems of musical ergonomics based around the requirements of tonal music and keyboard performance [12]. This work is reflected in the isomorphic keyboard design. Further evaluation of these features was outside the scope of this research.

The Playability Problem
The question of "playability" in the context of interfaces for musical expression has been approached from a number of angles [3,9,22]. The totality of the problem is quite complex and includes physical and cultural dimensions. The diagram below indicates the cognitive factors applied in musical performance, mediated by experience, skill, knowledge, memory, expectation, and the affordances of the instrument. The experimental design developed for this project provided the ability to capture control data before and after the mapping stage of a simple physical modeling instrument. This quasiexperimental approach enabled the observation of the influence of these various factors. The pilot study described below provided data that will used to reassess the issues raised by Hunt [9] with respect to usefulness of raw controller data in comparing the playability of various mappings. It is hoped that a model may be derived which by applying a least-squareregression algorithm will compensate for the divergence in qualitative judgments and simple quantitative estimations of playability.

Participants
Ten participants, all male, completed two simple sound matching tasks using the Thummer instrument. The participants had on average 15.5 years of musical experience. Only two of the participants played only one instrument. The average number of instruments played regularly by the participants was three. All participants indicated that they were regular computer users and had experience with electronic music production. It was later noted that regular use of computer game controllers appeared to have an influence on the participant's performance in the tests, but this information was not systematically collected.

Materials
The Thummer prototype was augment with a Microstrain 3DM-G 3-axis orientation sensor [11]. Only the left-hand joystick and orientation sensor were used in the experiment.
A patch was written in Max/MSP [23] to capture time-stamped control data from the Thummer instrument. The patch also included two simple mapping systems for the control of a physical model [4] of a bowed string instrument. The first mapping scheme used the X and Y-axes of the joystick, and the roll of the orientation sensor to control bow pressure, bow velocity and frequency offset (over a range of about 5%) respectively.
The second mapping scheme used change in the yaw over time (angular velocity) to control bow velocity. Pitch to control bow pressure and roll to control frequency offset.
A MIDI footswitch was used by the participant to initiate and conclude the recording of the data.
The captured data was subsequently analysed using Matlab software [10].
The experiments were videotaped and participants completed a short questionnaire relating to their musical experience.

Figure 9 Thummer in use showing orientation sensor
The system was calibrated for spatial orientation and the range of the joystick controls.
Participants were given a brief explanation of the first control system using diagrams and a practical demonstration. The participant was then allowed to explore the instrument and given several simple exercises to familiarize them with the instrument, the timbre and the physical control ranges. Given the simple nature of the control system and the lack of pitch or melodic possibilities, this process lasted an average of eight minutes.
The participant was then played the target sound, which had a duration of two seconds and featured a regular change in all three parameters over the duration of the sound resulting in a smooth amplitude envelope rising and falling as the pitch and brightness of the sound followed a similar contour. The target sound was repeated four times and then repeated once prior to each of four attempts during which the participant tried to match the sound based on listening only. No visual feed back was provided apart from that presented by the instrument itself.
The second half of the experiment repeated the procedure described above, using the second mapping scheme. Training times for the second half averaged seven minutes. Given that the participants were already familiar with the available range of timbres and the concept of the instrument and synthesis model this indicates a slightly longer time required to master the instrument.

Analysis
This Stage of the project set out to answer the following questions: 1) What are the useable number and range of control parameters offered by the prototype Thummer™? This question defines the playable range or gesture space of the prototype Thummer™?
2) What influence do control-mapping strategies have on the players ability to complete a simple performance gesture task?
3) What factors influence the "playability" of these prototype control mappings, from the performer's perspective?
Control data from the orientation sensor was calibrated to provide angles of orientation. In normal hand-held use, the pitch is limited by the articulation of the wrist, elbows and shoulders to a range of about 200 degrees. The pitch angle parameter exhibits a range of about 100 degrees during the experiments. In this implementation, the floating-point data from the sensor was reduced to 7-bit integers for compatibility with MIDI controller data. While the full 360 degrees of yaw can be achieved in normal performance by articulating the wrist, arms and rotating the body, the algorithm used to capture angular velocity was optimized to give a full 7-bit range for a comfortable range of gestures. The maximum range captured during the experiment was 68/128 against a target range of 100/128. The maximum roll angle that can be achieved while holding the instrument with two hands is about 180 degrees. However, it should be noted that the range for each angle is dependent on the absolute angle of the other axes. For example, with the wrists flexed to their maximum vertical deflection the range of roll available in the wrists is limited to about 40 degrees.
It is likely that intensive use with any given mapping will reveal aspects of the available gesture space not considered during the design phase. This development of "extended technique" is characteristic of the use of all musical instruments in contemporary performance.
Treating the mapping selection as an independent variable in this quasi-experimental design, changes in the closeness of fit between target control data and that recorded during the experiments was observed. Calculating relative mean-square error, time difference, accuracy and precision values for each participant and each mapping created simple measures for playability. Predictably, the joystick mapping showed much higher scores for each of these measures for all subjects except one. Precision represents the repeatability of the control gestures across attempts. It is propose that low accuracy coupled with high precision scores may indicate factors outside the mapping as indicated in figure 7. This effect may be attributed to perceptual factors related for example to auditory perception. Further work is intended in extracting features from the data. It is proposed that further modeling may reveal a method to match the quantitative playability scores with subjective evaluation of the performance as discussed above.
Several observations were made regarding the variability in participant's performance. The most significant factor would appear to be the experiences the player brings to the instrument. This is likely to have more impact on an experiment with a limited training period than the results of continued practice. It was noted that the only string player was almost immediately able to produce sustained sounds smoothing out the fluctuations at the end of each "bow stroke" whereas others (including the researchers) were rarely able to achieve this effect. One participant showed very subtle control of the joystick almost immediately. The participant revealed that he was a computer game enthusiast who spent more time gaming than playing his instrument. The joysticks on the Thummer are of a similar form and operated with the thumbs in the manner of a typical game controller. It was found that the foot pedal used to initiate recording caused some participant's considerable difficulty and others none at all. This was attributed to the relative experience of pedal use for keyboard or guitar performance. The pedal was chosen as it was felt that this control would be less likely to influence performance than the use of the Thummer keyboard itself.
All subjects agreed that the second orientation sensor based configuration was more musically satisfying and showed the greatest potential. Four of the participants indicated that they considered this configuration easier to play although the results do not correlate with this observation.
In conclusion, the sample size in this pilot study does not support generalisable claims but it has provided a basis for useful approaches to evaluating playability that can be pursued in future work.

Conclusion
The ThuMP project commenced with performance practice analysis of highly experiences players of exceedingly successful acoustic instruments. The project sought to ground itself therefore in centuries of instrument design and performance practice, and to leverage this knowledge as the foundation for mapping strategies for the Thummer instrument.
Qualitative analysis outcomes proved initially difficult to defined, however the change to the NVivo tool and the support of the third author provided useful detailed taxonomic diagrams of control parameter inter-relationships which formed the basis of the second round of interviews and the resultant mapping strategies for the Thummer and the usability studies.
The mapping strategies used in the playability testing were kept to simple single voice sounds in order to focus on the mapping relationships and the link between mapping and morphology of the sound.
The design recommendations hint at generalized design considerations to which the NIME community may like to contribute further.
Thumtronics continues to develop the Thummer instrument/interface with a projected release date in 2008.