Report Open Access

DeepCrime: Mutation Testing of Deep Learning Systems Based on Real Faults

Nargiz Humbatova; Gunel Jahangirova; Paolo Tonella

Deep Learning (DL) solutions are increasingly adopted, but how to test them remains a major open research problem.  Existing and new testing techniques have been proposed for and adapted to DL systems, including mutation testing. However, no approach has investigated the possibility to simulate the effects of real DL faults by means of mutation operators.

We have defined 35 DL mutation operators  relying on 3 empirical studies about real  faults in DL systems. We followed a systematic process to extract the mutation  operators from the existing fault taxonomies, with a formal phase of conflict resolution in case of disagreement. We have implemented 24 of these DL mutation operators into DeepCrime, the first source-level pre-training mutation tool based on real DL faults.  We have assessed our mutation operators to understand their characteristics: whether they produce interesting, i.e., killable but not trivial, mutations. Then, we have compared the sensitivity of our tool to the changes in the quality of test data with that of DeepMutation++, an existing post-training DL mutation tool. 

Files (789.0 kB)
Name Size
TR-Precrime-2021-06.pdf
md5:05763ee9f6ff44b55de5c4b62d473900
789.0 kB Download
47
169
views
downloads
All versions This version
Views 4747
Downloads 169169
Data volume 133.3 MB133.3 MB
Unique views 4343
Unique downloads 165165

Share

Cite as