The White Box Paradox in Medicine, Its Discontents and Possible Solutions

As cases of worsened automation bias after being presented with system explanations have been reported in specialized literature, the optimal timing of explanation provision during the human-AI interaction should be carefully designed and investigated in order to avoid anchor effects, which hinder an independent evaluation of the case at hand. In our experimental design, we took into consideration whether AI advice and explanation were provided either before the human decision (so called “ram” protocols) or after (“hound” protocols). We applied these two interaction protocols in an empirical study on the task of abnormality detection in knee magnetic resonance imaging, augmented by relevance map. In that occasion, we experimentally observed that visual explanations may paradoxically make users less confident in the final decision. We also described the white box paradox, whereby explaining the decisions of a black-box algorithm may make it a less useful or more harmful support. We therefore propose a series of design solutions to remedy these drawbacks by adopting the anti-agential perspective of adjunction: the machine is marginalized in the loop of human action and decision, with the aim of providing support to humans without affecting their autonomy, agency, and accountability towards those impacted by their decisions.