Published January 1, 2019 | Version v1
Report Open

An Empirical Validation of Oracle Improvement

  • 1. Università della Svizzera italiana
  • 2. University College London
  • 3. Facebook and University College London


We propose a human-in-the-loop approach for oracle improvement and analyse whether the proposed oracle improvement process is helping developers to create better oracles. For this, we conducted two human studies with 68 participants overall: an oracle assessment study and an oracle improvement study. Our results show that developers exhibit poor performance (29% accuracy) when manually assessing whether an assertion oracle contains a false positive, a false negative or none of the two. This shows that automated detection of these oracle deficiencies is beneficial for the users. Our tool OASIs (Oracle ASsessment and Improvement) helps developers produce assertions with higher quality. Participants who used OASIs in the improvement study were able to achieve 33% of full and 67% of partial correctness as opposed to participants without the tool who achieved only 21% of full and 43% of partial correctness. 



Files (1.1 MB)

Name Size Download all
1.1 MB Preview Download

Additional details

Related works

Is obsoleted by
10.1109/TSE.2019.2934409 (DOI)


PRECRIME – Self-assessment Oracles for Anticipatory Testing 787703
European Commission