Search-based DNN Testing and Retraining with GAN-enhanced Simulations

Our DESIGNATE approach supports the testing and improvement of vision DNNs by generating realistic but synthetic inputs belonging to distinct situations. Such realistic inputs are generated by combining meta-heuristic search, simulators, and GANs. Simulators enable the cheap generation of labeled input images (compared to labelling real-world images), while meta-heuristic search enables the exploration of the input space (i.e., exercise distinct situations). Finally, GANs improve the fidelity of the generated images, thus maximizing the likelihood that testing results are representative of what observable with real-world inputs, and, in turn, ensuring that failures observed during testing are due to limitations observable in the real-world and not to the fidelity gap of simulators. The images generated by DESIGNATE belong to distinct situations that are unsafe (i.e., the DNN poorly performs) and, further, can be used to retrain the DNN under test to improve its performance with real-world inputs.