Mouthbrush: A Multimodal Interface for Sketching and Painting

We present a novel multimodal input method permitting users to draw or paint using coordinated gestures of hand and mouth. A head-worn camera and com- puter vision algorithm provides a noninvasive, comfortable and low-cost user inter- face. Combining colour thresholding and image morphology techniques permits seg- mentation of the region of the open mouth. An adaptive algorithm, based on Fisher’s discriminant, automatically determines colour and intensity thresholds to separate the inside of the mouth from surrounding skin regions. We have designed two new con- trollability measures, the circle control task and the line width control task which we use to compare the users’ performance using the Mouthbrush with their performance using previously existing input devices: a table with pressure-sensitive stylus and a stylus with a finger wheel. The results of the user studies show that the Mouthbrush offers advantages over the pressure-sensitive stylus and finger wheel controller. Tri- als of the Mouthbrush system by professional and amateur artists show that it is ready for use in creative artistic expression.