Replication Package for ML-EUP Conversational Agent Study

This is the replication package of the paper How to Support ML End-User Programmers through a Conversational Agent, published at ICSE 2024.

Replication Package Files

• Readme.pdf: document that describes the replication package and indicates how to use it. 

• 1. Forms.zip: contains the forms used to collect data for the experiment.

• 2. Experiments.zip: contains the participants’ and sandboxers’ experimental task workflow with Newton.

• 3. Responses.zip: contains the responses collected from participants during the experiments.

• 4. Analysis.zip: contains the data analysis scripts and results of the experiments.

• 5. newton.zip: contains the tool we used for the WoZ experiment.

• Interactions.pdf: explains Figure 4 of the paper in detail by depicting the interactions of P4.

• TutorialStudy.pdf: script used in the experiment with and without Newton to be consistent with all participants.

• Woz_Script.pdf: script wizard used to maintain consistent Newton responses among the participants.

• Dockerfile: docker definition of newton-docker.tar.gz.

• newton-docker.tar.gz: docker image that contains both the tool and the analysis files.

• LICENSE: license file describing the license of data and code files.

1. Forms.zip

The forms zip contains the following files:

• Demographics.pdf: a PDF form used to collect demographic information from participants before the experiments

• Post-Task Control (without the tool).pdf: a PDF form used to collect data from participants about challenges and interactions when performing the task without Newton 

• Post-Task Newton (with the tool).pdf: a PDF form used to collect data from participants after the task with Newton.

• Post-Study Questionnaire.pdf: a PDF form used to collect data from the participant after the experiment.

2. Experiments.zip

The experiments zip contains two types of folders:

• exp[participant’s number]-c[number of dataset used for control task]e[number of dataset used for experimental task]. Example: exp1-c2e1 (experiment participant 1 - control used dataset 2, experimental used dataset 1)

• sandboxing[sandboxer’s number]. Example: sandboxing1 (experiment with sandboxer 1)

Every experiment subfolder contains:

• warmup.json: a JSON file with the results of Newton-Participant interactions in the chat for the warmup task.

• warmup.ipynb: a Jupyter notebook file with the participant’s results from the code provided by Newton in the warmup task.

• sample1.csv: Death Event dataset.

• sample2.csv: Heart Disease dataset.

• tool.ipynb: a Jupyter notebook file with the participant’s results from the code provided by Newton in the experimental task.

• python.ipynb: a Jupyter notebook file with the participant’s results from the code they tried during the control task.

• results.json: a JSON file with the results of Newton-Participant interactions in the chat for the task with Newton.

To load an experiment chat log into Newton, add the following code to the notebook:

import anachat
import json

with open("result.json", "r") as f:
    anachat.comm.COMM.history = json.load(f)

Then, click on the notebook name inside Newton chat

Note 1: the subfolder for P6 is exp6-e2c1-serverdied because the experiment server died before we were able to save the logs. We reconstructed them using the notebook newton_remake.ipynb based on the video recording.

Note 2: The sandboxing occurred during the development of Newton. We did not collect all the files, and the format of JSON files is different than the one supported by the attached version of Newton.

3. Responses.zip

The responses zip contains the following files:

• demographics.csv: a CSV file containing the responses collected from participants using the demographics form

• task_newton.csv: a CSV file containing the responses collected from participants using the post-task newton form.

• task_control.csv: a CSV file containing the responses collected from participants using the post-task control form.

• post_study.csv: a CSV file containing the responses collected from participants using the post-study control form.

4. Analysis.zip

The analysis zip contains the following files:

• 1.Challenge.ipynb: a Jupyter notebook file that performs the statistical tests and creates the perceptions of challenges figure.

• 2.Interactions.py: a Python file that creates the participants’ JSON files.

• 3.Interactions.Graph.ipynb: a Jupyter notebook file that creates the participant’s interaction figure.

• 4.Interactions.Count.ipynb: a Jupyter notebook file that counts participants’ interaction with each figure.

• config_interactions.py: this file contains the definitions of interaction colors and grouping

• interactions.json: a JSON file with the interactions during the Newton task of each participant based on the categorization.

• requirements.txt: dependencies required to run the code to generate the graphs and json analysis.

To run the analyses, please follow the steps:

1- Extract Analysis.zip and cd into the directory

2- Install Python 3.10, and then the analysis dependencies with the following command:

pip install -r requirements.txt

3- Run Jupyter Notebook/Lab and execute all cells of 1.Challenge.ipynb. It will generate the challenges figure.

4- Run 2.Interactions.py using the following command:

python 2.Interactions.py

This file was created manually by individually categorizing each interaction of the participants. The execution will generate the file interactions.json with the graph definitions of the interactions.

5- Run Jupyter Notebook/Lab and execute all cells of 3.Interactions.Graph.ipynb. It will create the interactions graph visualization.

6- Run Jupyter Notebook/Lab and execute all cells of 4.Interactions.Count.ipynb. It will create the interactions table.

5. newton.zip

The newton zip contains the source code of the Jupyter Lab extension we used in the experiments. Read the README.md file inside it for instructions on how to install and run it.

6. newton-docker.tar.gz

This file contains the Docker image with the replication package in a configured environment for both the tool and the dat analyses.

To import the image, run:

docker load < newton-docker.tar.gz

Then, start the container by running:

docker run -p 8888:8888 -it newton

Finally, start Jupyter Lab:

jupyter lab --collaborative --ip="*" --port=8888 --allow-root