Dear user, This README file is organized as follows: - In part (i), we explain in detail the different files and directories of the artifact. - In part (ii), we explain how to train our models, in the Unity3D environment. - In part (iii), we explain how to configure and compile the Marabou framework, which is the ``under the hood" SMT-solver that we used for our experiments. We also give precise instructions on how to run our original verification queries, as well as explain how to generate them from scratch with the supplementary scripts in this artifact. - In part (iv), we give instructions on how to run the gradient attacks used in our experiments. - In part (v), we present various CSV files summarizing the results of all our experiments. We note that the whole artifact (including the original models, our main code, and all experiments and results) --- are publicly and permanently available online. ######################################################################## (i) Files and Directories in the Artifact ######################################################################## Below is a summary of the contents of the artifact (in alphabetical order): ------------------------------------------------------------------------ 1. License.txt - a license for the use of our framework code. ------------------------------------------------------------------------ 2. models.zip: A compressed directory with all original 780 trained DRL neural networks. All the models are divided into one of 4 batches of experiments, corresponding to subdirectories ``batch_[i]'' for [i]=(1,2,3,4). Each batch includes the same number of models trained per algorithm: DDQN, REI (our abbreviation for ``Reinforce"), and PPO. The name of each subdirectory is in the format . For example, is a subdirectory with a frozen model trained via the DDQN algorithm, with a seed value of 47 and checkpoint (episode) number 26,095. All models are saved in TensorFlow 2 protobuf format. We note that for each seed there are 5 different episode checkpoints matching the seed. ------------------------------------------------------------------------ 3. property_encodings_for_marabou.zip A compressed directory with python scripts for encoding multiple k-step emulations of a DRL system, via the Marabou framework. This directory also includes scripts for encoding the six binary properties of interest (avoiding loops and collisions) mentioned throughout the paper. ------------------------------------------------------------------------ 4. summarizing_results.zip: A compressed directory with the CSV files summarizing all the results for our verification experiments and gradient attacks. This directory also includes a subdirectory with the results of our training phase. Each subdirectory (in the directory) corresponds to a different combination of seed and algorithm. ------------------------------------------------------------------------ 5. training.zip: A compressed directory with the python scripts for training the models. This directory contains three subdirectories and two files: (*) scripts - the implementations of the algorithms we used in our training (PPO, Reinforce, and DDQN). The third file is the parent abstract class for the reinforcement learning loop. The last file is the main class for the gradient-based attack. (*) env - a directory containing our simulator and the wrapper to run it as an OpenAI gym environment. The files are generated by the Unity3D engine with a 10x temporal speed value. (*) log - a directory that is (currently) empty, and will contain the logs and all the results of the training loops, when executed. (*) training.py - the main script for the training. The script require the seed as an input parameter to replicate our results. All original seed values are in the appendix of the paper. (*) gradient_attack.py - a script for running the gradient-based attack. The script is configured to run automatically on the saved models in the directory. ------------------------------------------------------------------------ 6. verification_queries_and_raw_results.zip: A compressed directory with two subdirectories: (*) bravery_property - including the raw results for the conservativeness (``bravery'') verification-driven analysis, described in the paper. (*) loops_and_collisions - including the raw results for all 6 binary properties checked throughout the model-selection process described in the paper. Specifically, the 3 collision cases and the 3 infinite loops. The directory also includes the original input query text files, each searching for a violation of a specific property in a specific model. ------------------------------------------------------------------------ 7. README.txt - this file. ------------------------------------------------------------------------ ######################################################################## (ii) Training Our Models ######################################################################## In this part, we explain how to train a model with our algorithms, and eventually save the best models (i.e., the models reaching a success rate that is equal/above our 96% cutoff). (*) the main script is: training/training.py To start a training loop, run the following command: (*) python training.py --algorithm --seed The two parameters are: 1. -> the algorithm to run, the accepted values are the keys for the algorithm: "PPO" (Proximal Policy Optimization) "DQN" (Double Deep Q-Network) "REI" (Reinforce) 2. -> the random seed for the training. This value must be an integer. In our experiment, we used values between 0 and 1,000. For example, a possible command to run Proximal Policy Optimization on with a seed of 3 is: (*) python training.py --algorithm PPO --seed 3 All the models and the CSV files summarizing the log of the training loop (Reward, Step, Success, ...) are automatically saved in the directory. The final models we trained are in TensorFlow 2 format. All the models and the CSV files with the logs of our training phase can be found in the and directories of this artifact. ######################################################################## (iii) Compiling Marabou & Running the Verification Queries ######################################################################## In this part, we will explain the 3 steps required to run the experiments described in our paper: 1. Downloading the Marabou framework 2. Compiling Marabou 3. Feeding the verification queries to Marabou We note that for ease of explanation, ``" (quotations) were added to describe code and subdirectories in the below description of files. As mentioned, Marabou is the under-the-hood SMT solver that we used for the experiments (other SMT solvers may be used as well). Please execute the following steps * Step 1: Downloading Marabou The Marabou framework is open-source, and can be freely downloaded in the following updated repository: https://github.com/NeuralNetworkVerification/Marabou * Step 2: Compiling Marabou Create a ``build" directory then build and compile Marabou. In Linux, run the following commands from the main artifact directory (after cloning it). Please open the file, and change line # 71: from -> const bool GlobalConfiguration::PREPROCESSOR_ELIMINATE_VARIABLES = true; to -> const bool GlobalConfiguration::PREPROCESSOR_ELIMINATE_VARIABLES = false; Then, build and compile the framework: ************************************************************************ cd Marabou mkdir build cd build cmake .. cmake --build . -j 8 ************************************************************************ This will create a ``build" subdirectory, and build and compile Marabou. We note that the compilation process itself may take time, due to running multiple tests for different aspects of the code. * Step 3: Feeding verification queries to the SMT solver Next, we can run the Marbaou SMT solver on each of the verification queries. We note that the original verification queries are supplied in an ``input query'' text file in the following subdirectories: for [i] in (1,2,3,4) representing the arbitraty batch number, and for any subdirectory in the format . All these subdirectories include 6 different subdirectories, each representing the verification queries (and results) for the 6 properties mentioned: 3 (1-step) collisions (FORWARD/LEFT/RIGHT), 1 (2-step) alternating loop and 2 (12-step) full cycles (LEFT/RIGHT). Note - in this artifact: (#) the alternating loop is commonly referred to as ``simple loop'' (#) the full cycle is commonly referred to as ``left loop" or ``right loop" Each of these 6 subdirectories includes 2 text files: 1. a query file (ending with ``ipq'') - encoding the specific verification query, searching for a violation of a single property, for a given model (in a batch of experiments). 2. an output file (ending with ``.output'') - encoding the results for the verification procedure, when feeding the predefined input query into the Marabou engine. These verification ``input query'' text files can later be fed into the verification engine, which outputs one of the following results: (*) UNSAT (*) MEMORY-OUT, in case the query was run on a machine with modest resources. (*) SAT, in which case Marabou will also return a satisfying assignment for the input. We note that, in accordance with common practice, we encode the *negation* of the *wanted* (correct) property. Thus, receiving SAT means the wanted property does *not* hold (and the engine supplies an input causing the bug). Thus, receiving UNSAT indicates that the property holds for any valid input. In order to run Marabou on a specific input query, we run the SMT solver via the cpp interface in the following way from the (unzipped) ``Marabou" directory (after building and compiling): ************************************************************************ cd Marabou/build ./Marabou --input-query=PATH_TO_QUERY ************************************************************************ For example, running (after building and compilation) the following code: Marabou/build/Marabou --input-query=verification_queries_and_raw_results/loops_and_collisions/batch_1/DDQN_id67_ep26871/left_collision_k_1/DDQN_id67_ep26871_left_collision_k_1_ipq will run the verification engine on the query searching for a left collision on the DDQN-trained model, with seed [67] and episode [26,871]. The raw outputs received for all the queries can be found in the ".output" files in the relevant subdirectories. We also refer the user to the directory, that includes 4 python scripts that (given Marabou is installed) load the maraboupy interface and encode an object with the relevant constraints. These scripts can be used to encode additional constraints or different multi-step properties, and then generate the input query text files, before feeding them to the Marabou verification engine. We also refer the user to the subdirectory, that includes (within each subdirectory of every model), an ``output'' file summarizing the binary search verification procedure run on each model, in order to assess its level of conservativeness (``bravery''). We note that unlike the case with the loop & collision properties, the input queries for each binary search were not predefined, due to each iteration of the binary search depending on the dynamic results from the previous iterations. For example, the output text file - verification_queries_and_raw_results/bravery_property/PPO_id73_ep12461/PPO_id73_ep12461.output represents a summary of all the verification queries sent, when conducting the binary search to rate the conservativeness level of the PPO-trained model with seed [73] and episode [12,461]. Note - for all verification experiments the slack was based on the 75th percentile of the empirical distribution matching each of the 3 algorithms' differences between the winning label and the runner-up (2nd highest label). Also, for each query, the slack was divided by the number of steps (k) that are emulated per each property. For more details please see the appendix of the paper. ######################################################################## (iv) Gradient Attacks ######################################################################## In this section, we explain how to run the gradient-based attack on the trained models. The results of the analysis are reported step by step after each attack. The main script for the procedure is: (*) gradient_attack/training.py To start the attack, run the following command: (*) python gradient_attack.py --algo --slack The two parameters are: 1. -> the algorithm on which run the analysis, the accepted values are the keys for the algorithm: "PPO" (Proximal Policy Optimization) "DQN" (Double Deep Q-Network) "REI" (Reinforce) The scripts search for the models to analyze in the directory, and subsequently select all the models with the key in their name. 2. -> the value of the slack variable. The attack runs, as explained in (iii) and in the main paper, with a fixed slack variable. In order to replicate the results of the paper, the suggested values for the different algorithms are: (*) DQN: 0.042 (*) REI: 1.904 (*) PPO: 4.821 The differences among the slack variables are due to the various ranges of the models, which depend on the selected training algorithm. An explanation regarding the chosen slack values (per algorithm) appears in the supplied appendix of our paper. The full results of our experiments can be found in the ``summarizing_results" directory. ######################################################################## (v) Experimnts - Summary & Results ######################################################################## The results of our experiments can be found in the directory. Below is a summary of the various files: (*) batches_1_to_4_full_property_results.csv - A CSV file summarizing the results of the verification process on all the trained models, and all 6 binary properties for each mode. The file includes listings of each model (batch/type/seed/episode) and additional columns indiacting the type of query (``query_type'', ``direction'' and ``k_steps''), as well as a string (in the rightmost column) indicating the result for the given query - SAT / UNSAT / T.O. / FAILED. (*) batches_1_to_4_full_bravery_results.csv - A CSV file summarizing the results of the verification process on all the trained models, and the conservative (``bravery'') experiments. The file includes listings of each model (batch/type/seed/episode) and a column indicating whether the binary search was successful (``successful_binary_search''). In cases where the search was successful, the rightmost column (``highest_bound'') includes the highest lower-bound (up to a given precision) rendering the query UNSAT, when the central lidar input is between the fixed lower-bound and the ``highest_bound'' value. For more details see the appendix of our paper. (*) batches_1_to_4_all_experiments_combined.csv - A CSV file combining the results from the two previous CSV files. Each row includes a single model, and the columns indicate the result of all properties verified. (*) training_output - A subdirectory containing the output files summarizing our training phase. This directory contains a subdirectory for each seed with the best models we found (i.e., the ones with a success rate above our high cutoff value). All the models are in the .h5 format and can be analyzed with the gradient-based attack as explained in section (iv). Each of these subdirectories contains also a CSV file with the log of the training loop. (*) gradient_attack.csv - A single CSV file containing the results of the gradient-based attack on the best models we found in our training (780 models total). Notice that the gradient attack is a stochastic procedure, and thus running multiple executions on the same input, may lead to different results. ------------------------------------------------------------------------ Thank you for your time! Guy Amir Davide Corsi Raz Yerushalmi Luca Marzari David Harel Alessandro Farinelli Guy Katz