Testing Scratch Programs Automatically

# Replication Package

This is the replication package for our work on
"Testing Scratch Programs Automatically".

The package contains our raw results and scripts for generating
the plots of the paper from the raw data.

## Abstract

Block-based programming environments like Scratch foster engagement
with computer programming and are used by millions of young learners.
Scratch allows learners to quickly create entertaining programs and
games, while eliminating syntactical program errors that could
interfere with progress.

However, functional programming errors may still lead to incorrect
programs, and learners and their teachers need to identify and
understand these errors. This is currently an entirely manual process.

In this paper, we introduce a formal testing framework that describes
the problem of Scratch testing in detail. We instantiate this formal
framework with the Whisker tool, which provides automated and
property-based testing functionality for Scratch programs.

Empirical evaluation on real student and teacher programs
demonstrates that Whisker can successfully test Scratch programs,
and automatically achieves an average of 95.25% code coverage.

Although well-known testing problems such as test flakiness also
exist in the scenario of Scratch testing, we show that automated and
property-based testing can accurately reproduce and replace the
manually and laboriously produced grading efforts of a teacher, and
opens up new possibilities to support learners of programming in
their struggles.

## Contents

The replication package is structured into two main directories:

* 'data/':
    raw data and scripts that have been used for collecting the data

* 'scripts/':
    scripts for generating the plots that are presented in the paper

### RAW data

* 'data/teacher-data/'
    data from the scratch workshop: sample solution and scores for student solutions

* 'data/code-club-stats/'
    block counts and input methods of the used Code Club projects

* 'data/coverage`
    code for measuring the coverage of automated input generation

* 'data/coverage-results/'
    coverage measurements on the Code Club projects

* 'data/test/'
    test suites for the projects of the Scratch workshop

* 'data/test-results/'
    test results from the test suites in 'data/test/'

* 'data/time/'
    Scratch programs for time measurement (10x the sample solution from 'data/teacher-data/`)

* 'data/time-results/'
    time measurements on the projects in 'data/time/'

## Reproducing the Plots

### Prerequisites

We describe the process based on:

    * the R statistics package in version 3.5
    * an Unix environment (Linux or MacOSX)

Following R packages are required:

    * ggplot2
    * dplyr
    * viridis

The package can be installed with the R command "install.packages".

### Generating the Plots

Coverage (Figure 10)

    ./scripts/coverage.R

    The result is a set of "coverage-*.pdf" files

Inconsistency (Figure 9)

    ./scripts/consistency.R

    The result is a set of "consistency-*.pdf" files

Scatter Plots (Figure 8, Figure 11)

    ./scripts/scatter.R

    The result is a set of "scatter-*.pdf" files