Risk-driven Online Testing and Test Case Diversity Analysis for ML-enabled Critical Systems (Replication Package)

This repository contains the replication package as well as obtained results for the study titled "Risk-driven Online Testing and Test Case Diversity Analysis for ML-enabled Critical Systems". These include simulation control script, simulation file, test output and evaluation scripts. 

Hint: [search algorithm] refers to ES, GA, SA or RS. For instance, cais_zmq_[search algorithm].py could be cais_zmq_ES.py or cais_zmq_GA.py etc.

• cais_scenario.ttt is the actual simulation file and is loaded into CoppeliaSim
• cais_zmq_[search algorithm].py is the control script for each search algorithm and connected via the ZMQ zmqRemoteApi package file
• clusterdata[search algorithm].csv contains the output from clustering all test evaluations for the respective algorithm
• result_evaluation.py is the evaluation script
• Distribution.pdf is related only to distribution of failed evaluations
• evaluation results.zip contains outcomes from test data analysis (grouped[0,1,2].pdf are boxplots for failed, false "positives" and passed evaluations respectively)
• README.md instructions to run replication package
• result_[search algorithm].txt all test evaluations per algorithm
• run_details_[search algorithm].csv objective function output and time taken for each test evaluation per algorithm
• vargha_delaney.py python module for Vargha Delaney effect size determination

Note: Instructions to run the package can be found in README.md