# Artifact Evaluation Package for POPL'24 Submission

This repository contains the artifact evaluation package for the POPL'24 submission titled "Validation of Modern JSON Schema: Formalization and Complexity" by Lyes Attouche, Mohamed-Amine Baazizi, Dario Colazzo, Giorgio Ghelli, Carlo Sartiani, and Stefanie Scherzinger.

All artifacts, including source code, data, and scripts, are bundled within a virtual machine running Ubuntu Linux. To access and run this virtual machine, please download it from Zenodo using the link provided:  [Zenodo, DOI: 10.5281/zenodo.10019663](https://doi.org/10.5281/zenodo.10019663). 

We recommend using VirtualBox to run the virtual machine. 

## Inspecting Individual Artifacts
For the convenience of reviewers, we provide detailed instructions on how to inspect and interact with each artifact separately. We refer to the sections below for specific guidelines.

## Contact 

[Contact Support: Stefanie Scherzinger](mailto:stefanie.scherzinger@uni-passau.de)








# List of Claims

Main claims in the article:

- Claim 1: Our formalization of Modern JSON Schema is correct.
- Claim 2: We have *formally* proved that validation in Modern JSON Schema is PSPACE-complete in the presence of dynamic references (Theorem 5, Corollary 6, and Theorem 7), while it is in PTIME when dynamic references are not present. 
- Claim 3: We have *formally* proved that validation is in PTIME when in the schema the maximum number of different fragments that are argument of "$dynamicRef" is bounded by a constant (Theorem 9 and Corollary 10).
- Claim 4: We have *formally* proved that, even in the presence of dynamic references, validation is in PTIME when the schema is fixed (Theorem 11 and Corollary 12).

In our experiments, we show the following regarding Claim 1: 
- Our validator correctly validates instances and schemas from the official JSON Schema Test Suite, which covers a wide range of language features.

In our experiments, we show the following regarding Claim 2: 

- Subclaim 2.1:  There exist families of schemas where the difference in validation complexity is reflected by considerable validation times.
- Subclaim 2.2:  This difference already manifests with small schemas.

# Download, Installation, and Sanity-Testing Instructions

1. Download the virtual machine (VM) image from Zenodo. 
2. Start up the VM image using VirtualBox. The VM was built using VirtualBox 7.0.
We recommend configuring 2 CPUs and 6 GB of main memory.
3. Log on at Ubuntu login screen (user `mjs`, password `mjsuser`).
3. Open a `xterm` terminal and proceed with sanity testing, described below.

## Sanity Tests

### Validator smoke test

Our validator *mjs-validator* is implemented in Scala. We wrote a Java harness to run our validator with *bowtie*,
a meta-validator for JSON Schema validators. This provides us with a uniform interface for invoking different validators.

First, ensure that the JSON Schema validator *mjs-validator* is installed and working,
by running a *bowtie* smoke test:

```bash
bowtie smoke -i mjs-validator
```

When successful, this returns:

```bash
Testing 'ghcr.io/bowtie-json-schema/mjs-validator'...
  ✓: allow-everything schema
  ✓: allow-nothing schema

✅ all passed
```

### Validating with *mjs-validator*


 Next, validate the instance `null` against a single schema that is universal, i.e., it accepts all instances.
 This instance is contained in the file `schemas/data.json`.

```bash
bowtie validate -i mjs-validator schemas/alternate.true.four/alternate.true.four.1.json schemas/data.json
```

In the output produced by *bowtie*, the string `"results": [{"valid": true}]` confirms that validation was successful (as expected).



# Evaluation Instructions

Our experiments validate the JSON instance `null` against three families of synthetic schemas, using different JSON Schema validators.
Each family of schemas encodes a formula, which we describe in the paper. 
Note that there is a 1:1 mapping between the names of the schema families in the paper to the schema files used in our experiments.
Reviewers can inspect the schemas [online](#data) and also look up the mappings.

In the VM, the schemas are stored in the directory `schemas`, the scripts are stored in the directory `scripts`.

Our dispatcher script evaluates the JSON instance `null` against schemas from these three families, using the *bowtie*-provided validators that support JSON Schema Draft 4 or Draft 2020.
In this comparative experiment, we also include our validator *mjs-validator*, as well as the academic JSON Schema validator *Jsch* that supports Draft 4.

The schemas and the academic validators are described in more detail further [below](#additional-artifact-description). 

## Phase 1: Kick the Tires

### Run Dispatcher Script as "Smoke Test"

The following command dispatches a limited version of our validation experiment, to show that the scripts are fully functioning.
The first parameter is a *label* that users can freely chose to label the experiment. Below, we simply chose `kick-the-tires`. The label helps identify the generated data,
and allows us to distinguish data generated in different runs.

The second parameter controls the number of schemas used in the experiment.
For now, we restrict ourselves to the first 5 schemas of each family, to limit the runtime.
Each single experiment is only repeated once, again to limit the runtime.


```bash
./scripts/dispatch.sh kick-the-tires 5 1
```


#### Duration
Running this experiment will approx. take one hour.


#### Evaluation

The generated data can be inspected in the `results` folder. 
There will be one directory for each individual experiment, and the label is used as suffix, so subdirectories which end in `kick-the-tires` contain our data.

In general, the directory name `results/validator.`_toolname_`.`_schemafamily_`.`_draft_`._label_ identifies the validator _toolname_,
the _schemafamily_ against which was validated, and the JSON Schema _draft_ (4 or 2020) used in validation.
The _label_ identifies the set of experiments.

The subdirectory `config` contains configuration information of the execution environment, including the hostname.

For each individual validation, two files are created. One contains the validation output, the other contains runtime measurements.

You may inspect these individual files, or generate a CSV summarizing the runtimes of each experiment with the following command.
Make sure you use the same input parameters as before.

```bash
./scripts/process_data.sh kick-the-tires 5 1
```

The CSV data is written to *stdout*, and also saved in the respective folders in the `results` directory.
Note that some validators produce runtime errors, so no data could be collected.
This is reported on *stdout*, and you will notice messages warning that no results exist for the validators `rust-jsonschema` and `s-vscode-json-languageservice`.


#### Inspect the Original Raw Data, Re-Generate the Charts

The VM image contains the results of the raw data from the original experiments, where 150 schemas from each family were validated, in 5 runs each:

You may inspect the directories named `results/validator.`_toolname_`.`_schemafamily_`.`_draft_`.03Jun2023` for the raw data.
Note that in the `config` subdirectory, you find the specifics of the original execution environment.
We also sketch the VM directory layout further [below](#vm-directory-layout).

You may also process the data into CSV files:

````bash
 ./scripts/process_data.sh 03Jun2023 150 5 
 ````

Finally, you can re-generate the charts. By passing the label `03Jun2023` you control which data to plot:

````bash
gnuplot -e "label_param='03Jun2023'" scripts/plot.plg
 ````

Inspect the generated pdf `plot.pdf` (e.g., using DocumentViewer) and visually compare with the original data in `plot_original.pdf`, or the article.


## Phase 2: In-Depth Evaluation 

We now discuss an in-depth evaluation, referring to the specific claims.

### Claim 1: Correctness

We applied our algorithm to the test cases of the official [JSON Schema test suite](https://github.com/json-schema-org/JSON-Schema-Test-Suite/tree/6afa9b38d84d45550ec703123eb4e8ec67a8ae75/tests/draft2020-12).
Since these tests are regularly updated, in order to account for reproducibility,
we must restrict ourselves to the tests for Draft 2020 in the specific version from commit hash 
`6afa9b38d84d45550ec703123eb4e8ec67a8ae75` from June 2023 (when we conducted our original experiments).

Below is the command to execute the tests in this version, using our validator *mjs-validator*.
Note that this command requires Internet access from within the VM.

````bash
bowtie suite -i mjs-validator https://github.com/json-schema-org/JSON-Schema-Test-Suite/tree/6afa9b38d84d45550ec703123eb4e8ec67a8ae75/tests/draft2020-12  | bowtie summary -s failures
 ````

If reviewers want to inspect the individual tests, they can request details with the command below:


````bash
bowtie suite -i mjs-validator https://github.com/json-schema-org/JSON-Schema-Test-Suite/tree/6afa9b38d84d45550ec703123eb4e8ec67a8ae75/tests/draft2020-12  | bowtie summary 
 ````

Note that our implementation passes all tests,
apart from 14 pertaining to schemas using the following characteristics: special characters in patterns,
references with an empty label or a label with special chars, unknown keywords, a vocabulary
different from JSON Schema, a decimal with a high precision. 
These limitations are also reported in the paper.

This experiment shows that the rules
that we presented, and which are faithfully reflected by our algorithm, are *correct and complete*
with respect to the standard test suite.


### Subclaims 2.1 and 2.2: Runtimes

To examine subclaims 2.1 and 2.2., repeat the experiment by calling the dispatcher with a label of your choosing (e.g. `artifacteval`),
the number of schemas to be used (here: 80) and the repetitions per run (here: 1): 

````bash
./scripts/dispatch.sh artifacteval 80 1
 ````



#### Duration
The original experiments were run on a 40-core Debian server with 384GB of
RAM, where we could easily exploit parallelism. Each core ran with 3.1Gz and CPU frequency set to performance mode. 

In bundling the artifacts in a VM image for evaluation, we did not assume that reviewers have comparable resources at hand.
Reviewers may therefore expect the full run of experiments to take *approx. 3 days*, depending on the CPU and memory resources.

#### Evaluation

Once the experiment has finished, collect the runtime data in CSV files:

````bash
 ./scripts/process_data.sh artifacteval 80 1 
 ````

After, generate new charts. Make sure to pass the matching label `artifacteval` so that you plot the right data:

````bash
gnuplot -e "label_param='artifacteval'" scripts/plot.plg
 ````

Inspect the generated pdf `plot.pdf` and visually compare with the plots in the article.

Expected observations:

The exact runtimes may differ between the target platform of the artifact evaluation and the original platform,
but the general runtime behavior should be evident:

- Plot `stati.js` (left) and `dyni.js` (middle): 
For these two families of schemas, our validator (magenta line) shows considerable difference in validation times.
(This confirms subclaim 2.1.)

- Plot `dyni.js` (middle): This difference already manifests with small schemas (below 10 on the x-axis). 

#### Caveats 

The original execution environment had 384GB of
RAM available. Assuming the hardware resources allocated to the Virtual Machine are more limited,
we must expect higher runtimes, and also fewer schemas that can be handled successfully.
In our tests of the VM, we were able to validate the `stati.js` schemas up until approx. schema `alternate.true.four.60.json`.
Nevertheless, the same general runtime behavior should be observable.


# [Additional Artifact Description](#Additional)

While our artifacts are all bundled within the VM, 
reviewers may want to individually inspect the artifacts. These are described below.

## Software

### bowtie

Our experiments rely on [**bowtie**](https://github.com/bowtie-json-schema/bowtie), a meta-validator for JSON Schema.
*bowtie* provides a common interface for invoking a collection of JSON validators. 
Our experiments were run with all 16 validators that support Draft4 and/or Draft2020 of JSON Schema,
and that were are available in June 2023. The git commit hash identifying the exact *bowtie* releases (for reproducibility) is [`1ef0cb8a8d5daf2ecbe9a154d7ce304225d298ce`](https://github.com/bowtie-json-schema/bowtie/releases).

### mjs-validator

We have packaged our prototype implementation called [**mjs-validator**](https://github.com/sdbs-uni-p/mjs-validator/tree/prebuilt) to be used with bowtie.
You may [inspect the Scala source code here](https://gitlab.lip6.fr/jsonschema/modernjsonschemavalidator).

### JSch validator

We have further packaged the Draft4 validator [**Jsch**](https://github.com/sdbs-uni-p/bowtie_JSch) for execution with *bowtie*.
This is an academic prototype by Felipe Pezoa, Juan L. Reutter, Fernando Suárez, Martín Ugarte, Domagoj Vrgoc, which is used in the experiments of the seminal paper ["Foundations of JSON Schema"](https://jreutter.sitios.ing.uc.cl/www16.pdf), published at WWW 2016.


## Data

In our experiments, we validate against three families of schemas written in the JSON Schema language.
These schemas have been designed to be universally satisfiable, so any JSON instance (such as `null`) validates against them.
The schemas are contained in the virtual machine, but for convenience, may also be inspected online at [https://github.com/sdbs-uni-p/mjs-schemas](https://github.com/sdbs-uni-p/mjs-schemas).

## VM Directory Layout

For reference, we outline the VM directory layout below.

```
├── schemas (contains schema files)
├── scripts (contains scripts for running/evaluating experiments, and plotting)
│
├── results (contains data produced in experiments)
│     ├── validator.cpp-valijson.alternate.true.four.4.03Jun2023 (data from original experiment)
│     │    ├── config (configuration data of the original run)
│     │    ├── alternate.true.four.1.json_run1_results  (output for the first schema)
│     │    ├── alternate.true.four.1.json_run1_time     (runtime info for the first schema)
│     │    │   ... 
│     │    └── alternate.true.four.results.csv          (CSV summarizing runtime data)
│     │ 
│     ...
│
├── plot_original.pdf (the plotted data from the original experiment, for reference)
└── plot.pdf          (the plotted data)

```

## Administration

The root password is `mjsuser`.

# Acknowledgments

We thank Stefan Klessinger and Sajal Jain for integrating the validators *mjs-validator* and *JSch* with *bowtie*.

*Last updated October 9, 2023.*