Report Open Access
An Empirical Validation of Oracle Improvement
Gunel Jahangirova; David Clark; Mark Harman; Paolo Tonella
We propose a human-in-the-loop approach for oracle improvement and analyse whether the proposed oracle improvement process is helping developers to create better oracles. For this, we conducted two human studies with 68 participants overall: an oracle assessment study and an oracle improvement study. Our results show that developers exhibit poor performance (29% accuracy) when manually assessing whether an assertion oracle contains a false positive, a false negative or none of the two. This shows that automated detection of these oracle deficiencies is beneficial for the users. Our tool OASIs (Oracle ASsessment and Improvement) helps developers produce assertions with higher quality. Participants who used OASIs in the improvement study were able to achieve 33% of full and 67% of partial correctness as opposed to participants without the tool who achieved only 21% of full and 43% of partial correctness.
| Name | Size | |
|---|---|---|
|
TR-Precrime-2019-04.pdf
md5:87dbe8881bfb209edf8c3504533da259 |
1.1 MB | Download |
| All versions | This version | |
|---|---|---|
| Views | 23 | 23 |
| Downloads | 20 | 20 |
| Data volume | 22.0 MB | 22.0 MB |
| Unique views | 20 | 20 |
| Unique downloads | 19 | 19 |
- Publication date:
- January 1, 2019
- DOI:
-
- Grants:
-
European Commission:
- PRECRIME - Self-assessment Oracles for Anticipatory Testing (787703)
- Related identifiers:
- Obsoleted by
- License (for files):
- Creative Commons Attribution 4.0 International