Report Open Access

DeepCrime: Mutation Testing of Deep Learning Systems Based on Real Faults

Nargiz Humbatova; Gunel Jahangirova; Paolo Tonella

Deep Learning (DL) solutions are increasingly adopted, but how to test them remains a major open research problem.  Existing and new testing techniques have been proposed for and adapted to DL systems, including mutation testing. However, no approach has investigated the possibility to simulate the effects of real DL faults by means of mutation operators.

We have defined 35 DL mutation operators  relying on 3 empirical studies about real  faults in DL systems. We followed a systematic process to extract the mutation  operators from the existing fault taxonomies, with a formal phase of conflict resolution in case of disagreement. We have implemented 24 of these DL mutation operators into DeepCrime, the first source-level pre-training mutation tool based on real DL faults.  We have assessed our mutation operators to understand their characteristics: whether they produce interesting, i.e., killable but not trivial, mutations. Then, we have compared the sensitivity of our tool to the changes in the quality of test data with that of DeepMutation++, an existing post-training DL mutation tool. 

Files (789.0 kB)
Name Size
789.0 kB Download
All versions This version
Views 7272
Downloads 445445
Data volume 351.1 MB351.1 MB
Unique views 6767
Unique downloads 425425


Cite as