The OVA File is a 64-bit Ubuntu. USERNAME : artifact PASSWORD : artifact Once logged in to the VM, the desktop contains the folder 'PRISM_analysis_courtrooms-main' which contains the script in the subdirectory 'Codes' and PRISM-related model and properties files under 'PRISM_Files' which further has the model inside 'Model' and the properties inside 'Properties'. The script is saved with the name 'PRISM_file_generator.ipynb' which is written in Python and is a jupyter notebook (both pre-installed). The code can be simply run and seen for the outputs. The code is separated into blocks where each block explains what it does. The output is a PRISM file of the model constructed. To build and verify the PRISM models generated by our code, PRISM model checker is already installed on the machine as well. PRISM is located at /Home/Downloads/prism-4.6-linux64/bin where prism is the command line tool and xprism is the GUI-based tool. DO NOT use the GUI-based tool for this artefact as the model is big and we would need to update the settings file of the GUI-based tool to achieve the same and also would be a time taking process. The expected execution time for the script is around 5-7 minutes. The time taken by PRISM to build the model is around 16 to 20 minutes. Model checking each property takes around 1-3 seconds and running experiments (which run over the entire state space) will take 1.5 to 2 hours. Since, models need not be constructed each time for model-checking a property, starting from our code, to the model being built once and then checked for 10 to 20 properties can be achieved in less than half an hour. We also suggest moving to a local machine for running the experiments if possible. Other details of the VM: Base Memory: 8192 MB Processors: 4 Video Memory 16 MB Reproducibility Instructions: The sub-directory 'Codes' contains a Jupyter Notebook which constructs the PRISM model from the pre-processed data sets (https://convokit.cornell.edu/documentation/supreme.html) of the two years of the transcripts of the Supreme Court of the United States (2018-19) which are stored as csv files named out1.csv and out2.csv respectively. The code is written in Python. The code outputs the PRISM model. There is no input from the user expected in the code. The code is separated across blocks with a description of what each block achieves. Just run through each window to get the desired PRISM model. The sub-directory 'PRISM_Files' contains the PRISM model in the 'Model' directory and the Properties discussed in the paper in the 'Properties' directory. The name of each query is saved as 'Query_i' where i refers to the query number in the paper. Note that, each query discussed in the paper may be a combination of multiple properties which need to be verified, evaluated or run on the model and their results as a whole are discussed in the paper. One can build the PRISM model and run the experiments using the PRISM model checker which is available at https://www.prismmodelchecker.org/ with all other necessary documentation nicely presented at https://www.prismmodelchecker.org/manual/. An example and some troubleshooting guidelines: To build the model and check for the i-th query: Step 1: Go to the PRISM directory. Open a terminal and reach the PRISM bin: ~/Downloads/prism-4.6-linux64/bin/ ----------------------------------------------------------------------------------------------------------------------- Step 2: Once you have reached the required directory, to check all the properties inside Query_i at once, you can: ./prism ~/Desktop/PRISM_analysis_courtrooms-main/PRISM_Files/Model/Court_room.prism ~/Desktop/PRISM_analysis_courtrooms-main/PRISM_Files/Properties/Query_i.props -javastack 1g where 1g is the new memory assigned to the java stack. If, in particular, you wish to check the property k in file Query_i, you can: ./prism ~/Desktop/PRISM_analysis_courtrooms-main/PRISM_Files/Model/Court_room.prism ~/Desktop/PRISM_analysis_courtrooms-main/PRISM_Files/Properties/Query_i.props -prop k -javastack 1g ------------------------------------------------------------------------------------------------------------------------ While running these queries as a whole or one at a time, there are some other types of properties discussed as well which involve the variable j. We refer to such properties as experiments and they need to be run slightly differently. Such properties exist for queries 1,9,10,11 and 12 of the paper. All of them contain the integer variable j and the variable j varies from 0 to 10098 with an increment of 1. Note that, experiment-type properties cannot be run as a whole and need to be run one at a time. All such properties are commented (implying they would not be evaluated in default). The details on running experiment-type properties are discussed below: Now, steps 1 and 2 work for all properties which do not include any variables. All properties involving a variable (which we refer to them as experiment-type properties) are commented in the Query files as they will otherwise affect the running of queries that do not have any variable. For verifying a commented property which has the variable say j, you need to follow the following steps: Step 1: Some of the query files contain a mixture of evaluation and experiment properties. You can open the query file and uncomment the experiment properties by removing the '//' prefix at the start of the line containing the property, and save it. The query files are located at ~/Desktop/PRISM_analysis_courtrooms-main/PRISM_Files/Properties/ --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Step 2: Let us assume that property k of Query_i is an experiment-type property, you can model check it as: ./prism ~/Desktop/PRISM_analysis_courtrooms-main/PRISM_Files/Model/Court_room.prism ~/Desktop/PRISM_analysis_courtrooms-main/PRISM_Files/Properties/Query_i.props -prop k -const j=s:u:t -exportresults results.txt:csv -javastack 1g In the above command, the k-th property in the file Query_i will be run as an experiment where the variable j appearing in the formula will be varied in between s and t with an increment of u at each iteration and the results will be exported as a csv file with the name results.txt. For our case studies, the only variable across all the experiments is j and the lower and upper bounds to be considered are 0 and 10098 with an increment of 1. Hence, it will look like ./prism ~/Desktop/PRISM_analysis_courtrooms-main/PRISM_Files/Model/Court_room.prism ~/Desktop/PRISM_analysis_courtrooms-main/PRISM_Files/Properties/Query_i.props -prop k -const j=0:1:10098 -exportresults results.txt:csv -javastack 1g An example: If the property filter(max, P=?[X(x=j)],(x=j)) in Query_1 needs to be verified, open the Query_1 file and remove the prefix '//' at the beginning of the line containing the formula and then save it. Now, this is the 12th formula in the file and hence, can now be run as: ./prism ~/Desktop/PRISM_analysis_courtrooms-main/PRISM_Files/Model/Court_room.prism ~/Desktop/PRISM_analysis_courtrooms-main/PRISM_Files/Properties/Query_1.props -prop 12 -const j=0:1:10098 -exportresults results.txt:csv -javastack 1g All snippets above assume that the model and the property are in the same directory, i.e., the prism bin and the output result is also saved at the same directory which is indeed the case. If that is not the case, this can be alternately done as follows: ./prism ~/path-to-directory/Court_room.prism ~/path-to-directory/Query_i.props -javastack 1g The same can be done for the results of experiments if you wish to export them to a different directory using ~/path-to-directory/Results.txt:csv