Empirical data on the usability of eyenalyzer

How did the data collection work?
The goal of the study is to get insights to the usability of the data analysis tool eyenalyzer. Hence the participants complete 6 data analysis tasks using eyenalyzer, while their eye movements are recorded by eye tracking. Note that the tasks are the same as in this conference paper and the corresponding data repository, but are broken down into 13 sub-tasks; each task targets one action page in eyenalyzer. After each task, the participants rate the difficulty of the task (on a 7-point Likert scale); after completing all tasks, they fill out a questionnaire on their demographic data as well as the two validated questionnaires System Usability Scale (SUS) and User Experience Questionnaire (UEQ). The submitted solutions are later checked for correctness; the answers to SUS and UEQ are transformed into scores. The manual rewatching of the eye tracking recordings gives the navgiation methods applied for switchting between the tasks or action pages of eyenalyzer as well as the handling of dialog boxes and info fields (i.e. to what extent they are read and whether or not the trigger a positive reaction or not). Further details on the procedure will be found in a soon to be published conference paper.

What data is provided?
In this repository you can find both the data obtained from 41 participants (i.e., folder data.zip) and a german replication package (i.e., folder material.zip). 

1. The data file data/performance.xlsx comprises 9 columns and 533 rows. Each row represents one combination of participant and sub-task. The columns are:
   • participant the current subject as a unique identifier from {P01, ..., P41}
   • task the current sub-task as one of {1A, 1B, 1C, 2A, 2B, 3A, 3B, 3C, 4A, 4B, 5A, 5B, 6}
   • score the grading of the current task for the current participant as one of {not completed, wrong, borderline, acceptable, correct}
   • time time used by the current participant for the current task in milliseconds
   • clicks the number of mouse clicks used by the current participant for the current task
   • keys the number of keys pressed on the keyboard by the current participant for the current task
   • difficulty difficulty estimate of the current participant for the current task as one of {very difficult, difficult, rather difficult, indecisive, rather easy, easy, very easy}
2. The data file data/questionnaire.xlsx comprises 18 columns and 41 rows. Each row belongs to one participant. The columns are:
   
   • participant the current subject as a unique identifier from {P01, ..., P41}
   • age age of the current participant in years
   • gender gender of the current participant as one of {male, female}
   • experience_programming previous experience of the current participant in programming as one of {none, little, indecisive, some, much}
   • experience_data_science previous experience of the current participant in data science as one of {none, little, indecisive, some, much}
   • experience_experimental_research previous experience of the current participant in experimental research as one of {none, little, indecisive, some, much}
   • experience_statistics previous experience of the current participant in statistics as one of {none, little, indecisive, some, much}
   • background the occupation of the current participant as one of {bachelor student, master student, phd student, job}
   • field the field of study or work of the current participant as one of {social sciences, computer science and mathematics, technology and engineering, economy and law, art and design and music, other}
   • SUS the SUS-score obtained by the current participant 
   • attractiveness the score on the UEQ-scale attractiveness obtained by the current participant
   • perspicuity the score on the UEQ-scale perspicuity obtained by the current participant
   • novelty the score on the UEQ-scale novelty obtained by the current participant
   • stimulation the score on the UEQ-scale stimulation obtained by the current participant
   • dependability the score on the UEQ-scale dependability obtained by the current participant
   • efficiency the score on the UEQ-scale efficiency obtained by the current participant
   • pragmatic_quality the mean score of the pragmatic UEQ-scales for the current participant (i.e. perspicuity, dependability, and efficiency)
   • hedonic_quality the mean score of the pragmatic UEQ-scales for the current participant (i.e. stimulation and novelty)
3. The data file data/navigation_method.xlsx comprises 3 columns and 246 rows. Each row represents one combination of participant and task. The columns are:
   • participant the current participant as a unique identifier from {P01, ..., P41}
   • task the current task as one of {T1, T2, T3, T4, T5, T6}
   • method the navigation method used by the current participant to switch to the proper action page for the current task as one of {navigation bar, landing page link}
4. The data file data/dialog_boxes.xlsx comprises 5 columns and 546 rows. Each row belongs to one dialog box triggering. The columns are:
   • participant the current participant as a unique identifier from {P01, ..., P41}
   • task the current sub-task as one of {1A, 1B, 1C, 2A, 2B, 3A, 3B, 3C, 4A, 4B, 5A, 5B, 6}
   • id the triggered dialog box as a unique identifier from {1, ..., 20}
   • reading the degree of reading of the current dialog box by the current participant as one of {yes, partial, no}
   • reaction the reaction on the current dialog box by the current participant as one of {good, bad, ignored, consecutive}
5. The data file data/info_fields.xlsx comprises 5 columns and 166 rows. Each row belongs to one info field triggering. The columns are:
   • participant the current participant as a unique identifier from {P01, ..., P41}
   • task the current sub-task as one of {1A, 1B, 1C, 2A, 2B, 3A, 3B, 3C, 4A, 4B, 5A, 5B, 6}
   • id the triggered dialog box as a unique identifier from {1, ..., 48}
   • reading the degree of reading of the current info field by the current participant as one of {yes, partial, no}
   • reaction the reaction on the current info field by the current participant as one of {good, bad, irrelevant}
6. The replication folder material.zip comprises three files and one subfolder with the following material:
   
   • primer.pdf information material on statistics provided to the participants
   • stimuli.pdf the stimuli used in the eye tracking set up (calibration and the work with eyenalyzer is denoted by dummy slides)
   • data/ a directory containing the 4 data files needed to complete the tasks (based on the data from this repository)
   • questionnaire.pdf a transcription of the demographic questionnaire

Do you have further questions?
For more information, please feel free to contact <lisa.grabinger@oth-regensburg.de>.