Supplemental for ChatGPT User Study - GenAI in Learning Software Engineering

Supplemental Material

Contents:

·       1-Demographic Information.xlsx: contains the demographic information of the participants in the study.

·       2-Forms.zip: contains the forms and questionnaires used to collect data for the experiment: demographic form, pre-study, post-study, and AAR/AI questionnaires.

·       3-GitHub-Repository.zip: a copy of the GitHub repository used in the study.

·       4-Tutorial Scripts.zip: script used in the experiment with the groups to be consistent with all participants.

·       5-Logs-Rubric-Grades.zip: contains the participant data log (commit and PR), rubric for grading submissions, and grades.

·       6-RQ1-Data-and-Analysis.zip: contains the data and analysis with respect to RQ1.

·       7-RQ2-Data-and-Analysis.zip: contains the data and analysis with respect to RQ2.

·       8-Participant Prompts.xlsx: contains the experimental group participant prompts with ChatGPT.

2. Forms.zip

The forms zip contains the following files:

·       Demographics.pdf: a form used to collect demographic information from participants before the study.

·       Control Pre-Study Questionnaire.pdf: Pre study questionnaire control group (Self-Efficacy Questionnaire)

·       Control Post-Study Questionnaire.pdf: Post study questionnaire control group (NASA-TLX, Self-Efficacy Questionnaire)

·       Treatment - AAR_AI task.pdf: Pre and Post task AAR/AI questionnaire for experimental group.

·       Experimental Pre-Study Questionnaire.pdf: Pre study questionnaire experimental group (Self-Efficacy Questionnaire, Question for Familiarity with AI)

·       Experimental-Post Study Questionnaire.pdf: Post study questionnaire experimental group (AAR/AI step 7, Continuance Intention, NASA-TLX, HAI Guideline Questions, Self-Efficacy Questionnaire)

3-GitHub-Repository.zip

The GitHub repository used in the study: contains the main.py code file and the Readme.md file (having the written instructions for the participants).

4-Tutorial Scripts.zip

Contains:

·       Control-Script.pdf: Script for the control group.

·       Experimental-Script.pdf: Script for the experimental group.

5-Logs-Rubric-Grades.zip

·       rubric.pdf: Created rubric for grading task performance.

·       GitHub-Task3-Log.xlsx: File containing the data regarding the status of commit made and PR raised for each participant.

·       grades.xlsx: Detailed grades for each participant in experimental (treatment) and control groups.

6-RQ1-Data-and-Analysis.zip

Note: The term 'treatment' has been used in the files of this folder to represent the experimental group: participants using ChatGPT for the tasks.

·       NASA TLX: folder containing the participant data (TLX.xlsx), code for statistical analysis (Stat-TLX.py) and statistical reports (analysis-TLX.csv).

·       Task Performance: folder containing the participant data (grades.xlsx & Scores.xlsx(overall grade)), code for statistical analysis (Stat-Correctness.py) and statistical reports (analysis.csv).

·       Self-Efficacy: folder containing:

              o   Self-Efficacy-detailed.xlsx: participant data

              o   Paired Stats: folder containing data (Total Self Efficacy.csv), code for statistical analysis(paired-stats.py),

                   and statistical reports (analysis.csv).

              o   Box plot: folder containing the code for generating the box plot and its output.

·       Continuance Intention.xlsx: participant data (experimental) for continuance intention of ChatGPT.

·       Stat-Table-H1-2-Paper.xlsx: Statistics table for NASA TLX and task performance as presented in the paper.

7-RQ2-Data-and-Analysis.zip

·       AAR_AI-Responses.xlsx: AAR/AI responses filled by participants in experimental group.

·       Quotation Manager-Faults&Conseq.xlsx: Contains the quotations from AAR/AI responses along with corresponding codes. Also contains the quotes that link faults to consequences in a separate sheet.

·       Codebook.xlsx: The final codebook (faults and consequences).

·       HAI-data.xlsx: Contains the reported guideline violations along with disaggregated analysis (grouped by gender).

·       Likert Plot-HAI: folder contains the code for generating the Likert plot figure presented in the paper.