How to Get Insight from Disagreeing Post-hoc Explanations in ML-based Defect Prediction Methods? An Empirical Study on Defect Predictors

Machine Learning (ML) based defect prediction models can help practitioners to identify bug-prone modules in large software projects. However, such defect predictors might not be accepted by practitioners due to the lack of interpretability. Therefore, post-hoc explanation methods such as LIME and SHAP have gained popularity. These explanation techniques offer insights into the decision-making of ML models by ranking features in order of importance. However, the post-hoc explainability of ML techniques is reasonably novel in the Software Engineering Community. Hence, it is unclear whether such methods help practitioners make better decisions regarding software maintenance. Furthermore, recent user studies show that data scientists often employ multiple post-hoc explainers to understand the decision of a single model due to the lack of ground truth. Such a scenario causes disagreement between explainability methods and prohibits drawing any conclusions. Therefore, our study first investigates disagreements among three explainers:  LIME, SHAP, and Breakdown. Second, we expose the more frequent disagreement types. Our finding leads to the proposition of an aggregation of post-hoc explanations that puts less emphasis on these disagreements while highlighting the aspect on which explanations agree. Thus, our aggregation method changes the ``perceived'' disagreement, which could help software engineers extract knowledge from multiple explanations.