Replication Package for ML-EUP Conversational Agent Study

Replication Package Files

• 1. Forms.zip: contains the forms used to collect data for the experiment

• 2. Experiments.zip: contains the participants’ and sandboxers’ experimental task workflow with Newton.

• 3. Responses.zip: contains the responses collected from participants during the experiments.

• 4. Analysis.zip: contains the data analysis scripts and results of the experiments.

• 5. newton.zip: contains the tool we used for the WoZ experiment.

• TutorialStudy.pdf: script used in the experiment with and without Newton to be consistent with all participants.

• Woz_Script.pdf: script wizard used to maintain consistent Newton responses among the participants.

1. Forms.zip

The forms zip contains the following files:

• Demographics.pdf: a PDF form used to collect demographic information from participants before the experiments

• Post-Task Control (without the tool).pdf: a PDF form used to collect data from participants about challenges and interactions when performing the task without Newton 

• Post-Task Newton (with the tool).pdf: a PDF form used to collect data from participants after the task with Newton.

• Post-Study Questionnaire.pdf: a PDF form used to collect data from the participant after the experiment.

2. Experiments.zip

The experiments zip contains two types of folders:

• exp[participant’s number]-c[number of dataset used for control task]e[number of dataset used for experimental task]. Example: exp1-c2e1 (experiment participant 1 - control used dataset 2, experimental used dataset 1)

• sandboxing[sandboxer’s number]. Example: sandboxing1 (experiment with sandboxer 1)

Every experiment subfolder contains:

• warmup.json: a JSON file with the results of Newton-Participant interactions in the chat for the warmup task.

• warmup.ipynb: a Jupyter notebook file with the participant’s results from the code provided by Newton in the warmup task.

• sample1.csv: Death Event dataset.

• sample2.csv: Heart Disease dataset.

• tool.ipynb: a Jupyter notebook file with the participant’s results from the code provided by Newton in the experimental task.

• python.ipynb: a Jupyter notebook file with the participant’s results from the code they tried during the control task.

• results.json: a JSON file with the results of Newton-Participant interactions in the chat for the task with Newton.

To load an experiment chat log into Newton, add the following code to the notebook:

import anachat
import json

with open("result.json", "r") as f:
    anachat.comm.COMM.history = json.load(f)

Then, click on the notebook name inside Newton chat

Note 1: the subfolder for P6 is exp6-e2c1-serverdied because the experiment server died before we were able to save the logs. We reconstructed them using the notebook newton_remake.ipynb based on the video recording.

Note 2: The sandboxing occurred during the development of Newton. We did not collect all the files, and the format of JSON files is different than the one supported by the attached version of Newton.

3. Responses.zip

The responses zip contains the following files:

• demographics.csv: a CSV file containing the responses collected from participants using the demographics form

• task_newton.csv: a CSV file containing the responses collected from participants using the post-task newton form.

• task_control.csv: a CSV file containing the responses collected from participants using the post-task control form.

• post_study.csv: a CSV file containing the responses collected from participants using the post-study control form.

4. Analysis.zip

The analysis zip contains the following files:

• 1.Challenge.ipynb: a Jupyter notebook file where the perceptions of challenges figure was created.

• 2.Interactions.py: a Python file where the participants’ JSON files were created.

• 3.Interactions.Graph.ipynb: a Jupyter notebook file where the participant’s interaction figure was created.

• 4.Interactions.Count.ipynb: a Jupyter notebook file that counts participants’ interaction with each figure.

• config_interactions.py: this file contains the definitions of interaction colors and grouping

• interactions.json: a JSON file with the interactions during the Newton task of each participant based on the categorization.

• requirements.txt: dependencies required to run the code to generate the graphs and json analysis.

To run the analyses, install the dependencies on python 3.10 with the following command and execute the scripts and notebooks in order.:

pip install -r requirements.txt

5. newton.zip

The newton zip contains the source code of the Jupyter Lab extension we used in the experiments. Read the README.md file inside it for instructions on how to install and run it.