On the Performance of Method-Level Bug Prediction: A Negative Result. Appendix

Abstract Bug prediction is aimed at identifying software artifacts that are more likely to be defective. Most approaches defined so far target the prediction of bugs at class/file level. Nevertheless, past research has provided evidence that this granularity might be too coarse-grained, thus reducing the usability of bug prediction in practice. As a consequence, researchers have started proposing defect prediction models targeting a finer granularity, particularly targeting methods, providing promising evidence that it is possible to operate at this granularity. Particularly, models based on a mixture of product and process metrics provided the best results. 

In this paper, we first replicate previous research on method-level bug- prediction using different systems and timespans. Afterward, based on the limitations of existing research, we (1) re-evaluate method-level bug prediction models more realistically and (2) analyze the whether textual features— previously shown as valuable sources of information for the evaluation of software quality (yet surprisingly unexplored in this research field)—can be exploited to improve method-level bug prediction abilities. Key results of our study include that (1) the performance of the previously proposed models, tested using the same strategy but with different systems/timespans, is con- firmed. However, (2) when evaluated with a more realistic strategy all the models show a dramatic drop in performance exhibiting results close to that of a random classifier. In addition, we find that (3) the contribution of textual features within such models is limited and unable to improve the prediction capabilities significantly. As a consequence, our replication and negative results indicate that method-level bug prediction is still an open challenge.