There is a newer version of the record available.

Published January 2, 2024 | Version v1

Assessing the impact of hints in learning formal specification: Research artifact

  • 1. ROR icon INESC TEC
  • 2. Faculdade de Engenharia da Universidade do Porto (FEUP)
  • 3. Universidade do Minho (UM)
  • 4. ROR icon Centro de Computação Gráfica

Description

This artifact accompanies the SEET@ICSE article "Assessing the impact of hints in learning formal specification", which reports on a user study to investigate the impact of different types of automated hints while learning a formal specification language, both in terms of immediate performance and learning retention, but also in the emotional response of the students. This research artifact provides all the material required to replicate this study (except for the proprietary questionnaires passed to assess the emotional response and user experience), as well as the collected data and data analysis scripts used for the discussion in the paper. 

Dataset

The artifact contains the resources described below.

Experiment resources

The resources needed for replicating the experiment, namely:

  • alloy_sheet_pt.pdf: the 1-page Alloy sheet that participants had access to during the 2 sessions of the experiment. The sheet was passed in Portuguese due to the population of the experiment.
  • alloy_sheet_en.pdf: a version the 1-page Alloy sheet that participants had access to during the 2 sessions of the experiment translated into English.
  • task_permalinks.txt: a list of example permalinks pointing to the tasks that participants had to solve in the 2 sessions of the experiment.

Collected data

Data collected in the simple one-factor randomised experiment in 2 sessions involving 85 undergraduate students majoring in CSE. The experiment was validated by the Specialised Committee for Research in the Social Sciences and Humanities of the University of Minho, where the experiment took place. Data is shared the shape of CSV files with a header row, namely:

  • task_data.csv: main data collected from task-solving in the 2 sessions of the experiment, namely:
    • participant identification: participant's unique identifier (ID) and the assigned treatment (column HINT, either N, L, E or D);
    • sociodemographic information: participant's age (AGE), sex (SEX, 1 through 4 for female, male, prefer not to disclosure, and other, respectively), and average academic grade (GRADE, from 0 to 20, NA denotes preference to not disclosure);
    • overall task resolution: task resolution data in the two sessions, namely variables productivity (PROD1 and PROD2, between 0 and 12 solved tasks) and efficiency (EFF1 and EFF2, between 0 and 1); the number of tries is also provided (TRY1 and TRY2), from which efficiency was calculated;
    • task resolution per domain model: the task resolution data from the 1st session but split by domain model (suffix A for the social network and suffix B for course management), which was used to discuss a possible threat to internal validity in Section 5 of the paper;
    • emotional response data: summarised emotional response data calculated from the PrEmo 2 surveys (EMO1 and EMO2, sum of differential in positive and negative emotions, ranging from -14 to 14, NA denotes failure to submit the questionnaire); the negative (NEG1 and NEG2, from -7 to 7) and positive differentials (POS1 and POS2, from -7 to 7) summed in that metric are also provided;
    • user experience data: summarised user experience data from the UMUX surveys (UMUX1 and UMUX2, as a usability metric ranging from 0 to 100).
  • emotional_data.csv: detailed data collected from the emotional questionnaire in the 2 sessions, namely:
    • participant identification: participant's unique identifier (ID) and the assigned treatment (column HINT, either N, L, E or D);
    • detailed emotional response data: the differential in each of the 14 measured emotions in the 2 sessions, half of which positive (Admiration1 and Admiration2, Desire1 and Desire2, Hope1 and Hope2, Fascination1 and Fascination2, Joy1 and Joy2, Satisfaction1 and Satisfaction2, and Pride1 and Pride2, either -1 if decreased, 0 if maintained, 1 if increased, NA denotes failure to submit the questionnaire), and half of which negative (Anger1 and Anger2, Boredom1 and Boredom2, Contempt1 and Contempt2, Disgust1 and Disgust2, Fear1 and Fear2, Sadness1 and Sadness2, and Shame1 and Shame2, also either -1, 0 or 1), used in the detailed discussion in Section 6 of the paper.

Analysis scripts

The analysis scripts required to replicate the results reported in the paper, namely:

  • analysis.r: An R script to analyse the data in the provided CSV file, the results of which are reported in the paper; each performed analysis is documented within the file itself.

Setup

Running the analysis scripts requires the installation of the R software. R is freely available for download at https://cran.r-project.org/mirrors.html, with binary distributions available for Windows, Linux and Mac. The required R libraries will be installed by the script if needed.

Usage

After installing the R software, just run the script through the command line to get the result of the statistical tests. In particular, for Linux and Mac, you can just run

Rscript analysis.r

and for Windows

Rscript.exe analysis.r

The data analysis results are reported in the terminal by default. For more details on running R, you can consult https://cran.r-project.org/doc/manuals/r-release/R-intro.html#Invoking-R.

Files

alloy_sheet_en.pdf

Files (346.9 kB)

Name Size
md5:a86a341635ac73b18f8c4a0b282df858
161.6 kB Preview Download
md5:085e862ab4c6be7de4ef5e5e4dfa9ada
163.0 kB Preview Download
md5:0a53c67db508b25917f381a5685d367a
7.4 kB Download
md5:5d6403f3f9317c0aeb3fd6a69c01a579
6.1 kB Preview Download
md5:06ce185d4645c20b36e77fcfb0e4441f
335 Bytes Download
md5:a527bb13191d129f565d2801eebf74f8
7.2 kB Preview Download
md5:2e82572e0b824ae7b8207248976be525
1.2 kB Preview Download

Additional details

Related works

Is supplement to
Conference paper: https://nmacedo.github.io/pubs/ICSE24.pdf (URL)

Funding

Fundação para a Ciência e Tecnologia
SpecRep - Constraint-based Specification Repair EXPL/CCI-COM/1637/2021

Dates

Collected
2023-03-24/2023-03-31
Data collection