A systematic computerized training program for using Sensory Substitution Devices in real-life

In the past decades, Sensory Substitution Devices (SSDs) have been widely used as a research tool to unravel the properties of the sensory brain. Although their rehabilitation potential is repeatedly demonstrated, SSDs were never widely adopted by blind individuals in everyday life, except for a few super-users’ cases. One reason explaining this gap is the lack of structured SSD training programs for everyday use. We thus developed an ambitious computerized SSD training program using the EyeMusic visual-to-auditory SSD and gathered 10 blind participants to test its efficiency. Participants were trained to identify pictures of real objects from different categories (e.g., furniture). For each category, we tested the performance of participants before training and again after 10 hours of dedicated training. The test included both trained and untrained stimuli. The 10 hours training program involved a combination of static stimuli and dynamic computer games and was individually adapted to participants’ learning pace. Initial results show that after training, participants achieved significantly higher accuracy rates in object recognition compared to baseline for trained and most importantly, for untrained objects from the same category. This further supports the suitability of SSDs in everyday life, and thus propels their adoption for this purpose.


I. INTRODUCTION
An estimated 1.3 billion people around the world are visually impaired, and among them 36 million are legally blind [1].Among the many assistive and rehabilitative technologies for visually impaired individuals that have been developed, we will focus here on Sensory Substitution Devices (SSDs), a family of non-invasive devices for visual rehabilitation, and more specifically on an SSD that we developed in our lab called the EyeMusic [2].The EyeMusic sonifies the visual image into what we term 'auditory soundscapes', preserving the exact shape, location and color of visual objects.
The SSDs rehabilitation potential has been repeatedly demonstrated in lab environments in a variety of tasks (e.g., recognizing body postures of an animated figure) [3].However, they have not yet been adopted by visually impaired individuals for everyday use, beside the rare cases of SSD super-users [4].One reason explaining this gap is the lack of structured SSD training programs for advanced users, tailored for real-life objects.Thus, we developed an ambitious computerized SSD training program using the EyeMusic visual-to-auditory SSD, asking 1-can SSD advanced users learn to identify real objects from common objects' categories (e.g., furniture) after a focused training program.And 2-if yes, will they be able to generalize their identification abilities to novel, untrained objects within the trained categories.

A. Equipment
The EyeMusic algorithm was used to convey visual information via audition, by using a column-by-column sweep line algorithm that scans an image from left to right and down sample it to a 30x50 pixels resolution.The x-axis is mapped to the time domain: pixels located on the left side of an image will be heard before pixels located on the right side.The y-axis is mapped to the frequency domain: the higher the pixel is located, the higher the musical note sonifying it.Colors (for now white, blue, green, yellow, red) are mapped to different musical instruments, while black is mapped to silence [5].

B. Participants
10 blind adults participated in this study (mean age 42±9; 5 females): 8 congenitally blind, 1 early blind, 1 late blind.All participants were extensively trained on the EyeMusic algorithm to identify drawn shapes but were naïve to realworld objects and were never exposed to the stimuli from the categories used in this study.This experiment was conducted in accordance with the Helsinki declaration and all participants signed their informed consent.

C. Experimental design
We used a pre-post training paradigm to test the efficiency of the EyeMusic real-world category training program.We collected real pictures belonging to five objects categories: Furniture, Body parts, Fruits & Vegetables, Animals, Clothes.For the purpose of testing, each category was divided into 3 sub-categories (e.g., the clothes category was divided into: (a) shirts; (b) pants; (c) dresses & skirts;).Each category contained 50-90 stimuli.During training, participants always heard one stimulus at a time.Stimuli were always presented in the center of the screen, (i.e., center of the visual field).Training occurred separately for each category.Training methods were individually tailored but always included both specific explanations on the stimuli in the trained category, together with SSD games on the same category.For each category, participants performed a baseline object recognition task and then repeated the same task after 10 hours of training.During the object-recognition task, participants heard EyeMusic stimuli belonging to the given category, and they had to assign the object to one of the 3 defined sub-divisions by pressing the corresponding button on the computer.Each stimulus could be heard up to 3 times.All tests comprised 66% trained stimuli and 33% untrained stimuli.Each test consisted of 27 stimuli, which were repeated in a random order for a maximum of 4 times, thus generating a maximal pool of 108 stimuli.However, to limit fatigue effects during data collection, each test ended automatically after 10 minutes.Thus, depending on the participant's reaction times, around 40-50 stimuli were heard by each participant in each test.

III. RESULTS
For 4 out of the 5 categories, the performance of participants significantly improved between pre-and posttraining (10 hours) tests (Fig. 1).Specifically, these were the categories of body-parts, clothes, fruits & vegetables and furniture (all P-values < 0.05 in an unpaired 1-tail t-test): Body parts: pre-50±3.98%[mean±SE], post-68±3.22%;Clothes: pre-42±7.62%,post-64±5.61%; Fruits & Vegetables: pre-52±3.34%,post-66±1.87%;Furniture: pre-48±2.59%,post-62±4.50%.For the Animals category: no significant difference between pre-(53±7.17%)and post-training tests (68±10.27%)was found (P = 0.12 unpaired 1-tail t-test), even though a trend for improvement in the post-training was observed (Fig. 1).Importantly, for all categories, participants performed in the post-test significantly above the chance level of 33%, including in the Animals category (all P-values < 0.005 unpaired 1-tail ttest).When considering all categories together, learning was significant between the baseline test (49±2.23%)and the 10 hours training test (66±2.29%;P < 0.000005 unpaired 1-tail ttest).Furthermore, no significant difference was found between trained and untrained stimuli (P = 0.06 unpaired 2-tail t-test).Also, specifically for untrained stimuli, participants' performance significantly improved between pre-and posttraining tests for the Clothes, Body-parts and Furniture categories (all P-values < 0.003 in an unpaired 1-tail t-test), though not for the Animals and Fruits & Vegetables categories (P = 0.07, P = 0.43 unpaired 1-tail t-test) Fig. 1.Average accuracy for each category.Orange colums depict the average accuracy rate in the baseline test for all subjects and blue colums depict the average accuracy rate in the test after 10 hours training, plotted separetly for all the tested categories.SE bars are added for each column.Asterisks depict significant differences between pre-post-tests.
IV. DISCUSSION Our results support the conclusion that a structured and focused training program on SSDs, can lead to successful recognition of real-life objects.Crucially, we also showed that participants were able to successfully recognize untrained objects belonging to a trained category.Furthermore, our results highlight the potential flexibility of such structured SSD real-life training: after learning the principles of the SSD algorithm, users could continue their training focusing on reallife categories of their interest.This is particularly promising when considering that users are often discouraged from using SSDs due to the length of SSD training programs.We showed that 10 hours of structured training are enough to become proficient in recognizing objects belonging to a given category.
Currently, we are continuing to systematically test the feasibility of SSD for real-life use by using a live camera to test the EyeMusic in real interactions with the environment as well as in virtual reality (VR) environment (e.g., reaching/grasping).Indeed, training on the EyeMusic in VR can allow blind users to train with the EyeMusic in all possible real-life situations (e.g., finding a crosswalk in a busy street), though with no risk of being injured during the training phase, namely, before learning has occurred.We are also planning to evaluate the efficacy of the EyeMusic SSD to convey intrinsically visual concepts (e.g., depth) to further test the similarity between SSD and visual perception.This latter aspect carries crucial implications for visual rehabilitation, suggesting the intriguing possibility of pairing SSD to novel methods for visual restoration thus maximizing their rehabilitative outcomes [6].