Replication Package for "Empirically Evaluating the Impact of Object-Centric Breakpoints on the Debugging of Object-Oriented Programs"

This repository provides the necessary materials to replicate the empirical experiment analysis presented in the paper "Empirically Evaluating the Impact of Object-Centric Breakpoints on the Debugging of Object-Oriented Programs". It includes instructions, data, and artifacts to enable reproduction of the analysis. Additionally, the repository contains PDF files with diagrams of the extracted data whose numbers were reported in the paper. The files are primarily in Markdown format, we recommend reading them with a Markdown renderer. For those who prefer a more traditional format, PDF versions are also available.

Design

Within the "design" directory, you will find the components of the experiment definition and the procedures used to transform and interpret the collected data. The directory is structured as follows:

Design

R-scripts

The "R-scripts" directory contains the scripts to reproduce the results of our experiment. To run the scripts, you must have R installed on your system.

#install.packages("effectsize")
#install.packages("ggplot2")
#install.packages("Hmisc")
#install.packages("lobstr")

> Rscript R-scripts/reproduction.r

Data

.
├── Feedback - Additionnal feedbacks on the perception of the tasks
│   ├── about-treatment-difficulty.md
│   ├── ammolite-control-then-lightsout-treatment.md
│   ├── grey-feedback.md
│   └── lightsout-control-then-ammolite-treatment.md
├── data.csv - Contains the extraction of all questions and answers
└── extracted-data
    ├── controls - The statistical tests results for the comparison of both tasks under the control condition                                           
    │   ├── Controls-full-statistics.txt
    │   ├── control-actions-distribution.pdf
    │   └── control-times-distribution.pdf
    ├── demographics - Contains the repartition of the participants per-task depending on the different demographic factors. Please refer to questions-data-mapping.csv for more details about the demographic questions each file refers to. 
    │   ├── Ammolite-education.csv
    │   ├── Ammolite-frequency.csv
    │   ├── Ammolite-jobs.csv
    │   ├── Ammolite-pharo-frequency.csv
    │   ├── Ammolite-pharo.csv
    │   ├── Ammolite-prog.csv
    │   ├── LightsOut-education.csv
    │   ├── LightsOut-frequency.csv
    │   ├── LightsOut-jobs.csv
    │   ├── LightsOut-pharo-frequency.csv
    │   ├── LightsOut-pharo.csv
    │   ├── LightsOut-prog.csv
    │   ├── code.frequency.csv
    │   ├── education.csv
    │   ├── familiarity.csv
    │   ├── job.position.csv
    │   ├── pharo.exp.csv
    │   ├── pharo.frequency.csv
    │   ├── program.exp.csv
    │   ├── total-jobs.csv
    │   └── total-xp.csv
    ├── experiment-feedback
    │   └── experiment-feedback.csv - Contains the answers given by participants to the post-experiment questions. The results are presented by number/frequency of apparition of the different possible answers.
    ├── post-task-feedback - Contains the answers given by participants to the post-task questionnaires. The results are presented by number/frequency of apparition of the different possible answers, after the control task and the treatment task.
    │   ├── ammolite-agree-disagree.csv
    │   ├── ammolite-help.csv
    │   ├── ammolite-yes-no.csv
    │   ├── lightsout-agree-disagree.csv
    │   ├── lightsout-help.csv
    │   └── lightsout-yes-no.csv
    ├── questions-data-mapping.csv - Maps the name of each CSV file with the questions asked after each tasks and at the end of the experiment
    └── statistics - Contains the results of the statical tests (normality, significance) performed on Ammolite and Lights Out 
    │   ├── Ammolite-full-statistics.txt - Control-Treatment comparison for the Ammolite task
    │   ├── LightsOut-full-statistics.txt - Control-Treatment comparison for the Lights Out task
    │   ├── Ammolite-demographics-statistics.txt - Control-Treatment demographics for the Ammolite task (only chi-squared p-values)
    │   └── Ammolite-full-statistics.txt - Control-Treatment demographics for the Lights Out task (only chi-squared p-values)
    └── tools-usage - Contains the tools usage frequency per task and the statistical tests checking for a difference in tools usage between control and treatment (with the Benjamini-Hochberg procedure for limiting the false discovery rate).
        ├── BenjaminiH-procedure-for-tools-usage-Ammolite.pdf
        ├── BenjaminiH-procedure-for-tools-usage-LightsOut.pdf
        ├── BenjaminiH-procedure-for-tools-usage.xlsx
        └── detailed-tools-usage.csv