Failure Sources in Machine Learning for Medicine—A Study

Machine learning (ML) inherently suffers from at least a small amount of inaccuracy. Typically, these errors are acceptable in trade for either speed to an answer or the ability to find an answer at all. For high consequence domains, such as medicine where a wrong diagnosis can mean the difference between catching a disease early or not or prescribing debilitating treatment when it may not be needed, certain kinds and types errors are less acceptable. In a study attempting to reproduce ML for medicine research, many difficulties are encountered. These difficulties highlight both the need for higher standards to achieve reproducible ML in general and especially when it comes to high-stakes domains. This paper explores some of those difficulties with a focus on the error sources and discussions about how they may be addressed.