Published January 1, 2021 | Version v1
Report Open

DeepCrime: Mutation Testing of Deep Learning Systems Based on Real Faults

  • 1. Università della Svizzera italiana


Deep Learning (DL) solutions are increasingly adopted, but how to test them remains a major open research problem.  Existing and new testing techniques have been proposed for and adapted to DL systems, including mutation testing. However, no approach has investigated the possibility to simulate the effects of real DL faults by means of mutation operators.

We have defined 35 DL mutation operators  relying on 3 empirical studies about real  faults in DL systems. We followed a systematic process to extract the mutation  operators from the existing fault taxonomies, with a formal phase of conflict resolution in case of disagreement. We have implemented 24 of these DL mutation operators into DeepCrime, the first source-level pre-training mutation tool based on real DL faults.  We have assessed our mutation operators to understand their characteristics: whether they produce interesting, i.e., killable but not trivial, mutations. Then, we have compared the sensitivity of our tool to the changes in the quality of test data with that of DeepMutation++, an existing post-training DL mutation tool. 



Files (789.0 kB)

Name Size Download all
789.0 kB Preview Download

Additional details

Related works

Is obsoleted by
Conference paper: 10.1145/3460319.3464825 (DOI)


PRECRIME – Self-assessment Oracles for Anticipatory Testing 787703
European Commission