Errors and error management in biomedical research

The authors put a much-needed focus on the validity of biomedical research results that has come under scrutiny. However, as they make clear, due to the complexity of the experiments involved, errors quite naturally occur frequently. A way of managing these errors is the “Laboratory Critical Incident and Error Reporting System” (LabCIRS), a software tool to record all incidents anonymously and to analyze, discuss, and communicate them. It has been adapted from the CIRS, used in the clinical world to improve patient safety in complex, fast-paced, and often understaffed settings.


Introduction
Errare humanum est -To err is human. Biomedical research, a human enterprise, is no exception in this regard. Ever more sophisticated methodologies probing how complex organisms function in health in disease invite errors on all levels -from designing experiments and studies to the collection of data and the reporting of results. The stakes are high, in terms of resources spent, and professional rewards to be gained for individuals. Up to now the public holds scientists, and in particular those aiming to improve human health, at very high esteem. Nevertheless, the exposure of spectacular cases of fraudulent and irreproducible research 1,2 and the realization that biomedical results appear to lack robustness as well as the scientific process rigor, have recently begun to undermine some of that public trust 3 .
Even within the research community there is growing concern that a considerable fraction of research is actually waste 4 -a notion that is reflected in the fact that a majority of scientists agrees that we are experiencing a significant 'reproducibility crisis' 5 . The search for causes, and potential remedies, has led to considerable introspection, science has turned its scrutiny upon itself 6 .
Many factors have been singled out which may be invoked to explain the current concerns about the validity of biomedical research results. Top of the list are exceedingly low statistical power (i.e. sample sizes are too small), as well as low internal validity 7 . Internal validity encompasses a number of quality factors, most of them related to bias. Bias can be eliminated, or at least controlled, by such measures as randomization, blinding, or prespecification of in-and exclusion criteria.
Clearly, the discussion is revolving around factors which negatively impact on the quality of research -and which may be remedied by structured measures to improve research quality 8 .
The potential contribution of errors to the disappointingly low level of reproducibility and predictiveness of biomedical research, and how scientists deal with these errors, has not yet been considered. This is highly surprising, as error management plays a central role in any structured approach to safeguard quality ('quality management'). It is also safe to assume that due to its multiple levels of complexity, errors must be quite frequent in biomedical research. In the following we explore the types of errors that might occur in biomedicine. We then propose and offer a simple tool to establish a mature error culture in biomedical research.

Systematic (device) errors
Such errors happen when a device is failing in its precision to measure in the desired range, or a complete device malfunction occurs. It is especially difficult to discover such failures if the device does not contain a self-check mechanisms for proper operational range or an internal standard. A prominent example are measuring pipettes operating in the microliter range. In microbiology or cell biology experiments, faulty volume measurements result in serious mistakes and directly lead to false outcome measures that often go undetected. Therefore, every research laboratory, however small or big they are, need to make an inventory of their devices and lab equipment which provides information regarding warranty status, maintenance cycles, routine checks, and possible calibration procedures. Corresponding event dates need to be listed showing when these tasks were performed last and when they are due. Setting up an automated reminder system can prove very helpful to prevent errors due to lack of calibration. In industry contractors or service agents from device makers usually take over all these tasks for a fee. Academic biomedical research laboratories usually do not posses sufficient funds to outsource maintenance for all devices. Therefore, it is vital that they identify all critical devices in the experimental process and ensure that all of them function properly prior to the experiment. Only then valid results can be obtained. When noticing that an uncalibrated or otherwise quality-compromised device is being used in an experiment, every researcher needs to note this in his or her laboratory notebook along with results and protocol used. Only then results can be further evaluated and by comparison a decision can be made whether to keep or discard the experiment.

Errors due to limitation of measurement
Many laboratory devices use changes in physical properties (light or ray emission, chemical reaction) of the analyte as proxies for the parameter under study, simply because these are easily detectable and quantifiable. However, there are certain measurements that rely on human judgment, such as color scale matching of pH paper. Another more common measure using human evaluation scales are behavioral scores for laboratory animals. Even though a common description exists, discrepancies in interpretation among lab personnel or misjudgments occur and are major contributors to the large variance of behavioral experiments.
One strategy is replacement of these measurements with objective assays, e.g. calibrated, electronic pH meters. Regarding scoring behavior, training and easily understandable, detailed standard operating procedures are a good starting point. Ideally video examples representing specific scores can not only help during training but also offer better recall during actual scoring. Every training should be followed up by blinded tests to confirm the validity of the practice. In addition, multi-lab comparisons for test procedures can help identifying ambiguity in protocol and therefore ensure reproducibility.

Protocol deviation
Protocol deviation is any non-compliance with an existing protocol, standard operating procedure (SOP), or work instruction. Another irregularity related to this category is known as "protocol drift", which is present when a protocol is executed without or only limited supervision or content checks. If this alteration does not result in immediate experimental failure or is otherwise noticeable, the practice becomes the norm and is most often only detected by accident, if at all.
To combat this error, several steps can be taken. The SOP can be accompanied by a mandatory checklist list containing key elements of the protocol in a concise manner which need to be checked, or certain data that need to be inserted. Another element concerns mandatory regular checks of the protocol content by a supervisor or other responsible person. Every protocol should contain information regarding this validity check, i.e. an expiration date. Most effective against protocol deviations are internal reviews or method audits.

Reporting errors
Reporting errors can be the result of insufficient or faulty documentation during an experiment. Post-experimental analytical errors, such as statistical errors, also belong to this group. Without access to the original data or the original documentation it is hard to detect such errors, especially when the publications have already been peer-reviewed. Retraction Watch is widely read in the community and by journalists. Importantly, the editors of Retraction Watch also point out commendable retractions. As stated above, to err is human, and to stand by one's errors, to expose and correct them are important elements of the self correcting quality of science. Hopefully, this will foster the development of an error culture in science.

Errors of yet unknown cause
Laboratory protocols often contain experimental controls or checkpoints in which verification takes place, or a comparison to a known standard. Despite adherence to the protocol and verification of all used reagents, the obtained result may not measure up to the standard. We here classify these errors as "of yet unknown cause". It is important to communicate these errors and to keep a record of them. If such an error happens more than once, an underlying systematic error must be suspected which requires further investigation until the source is identified and eliminated.
For instance, cell survival rate in the preparation of a cell culture suspension from neonatal rodent brains is a standard procedure that delivers an anticipated yield of living neuronal cells in culture when the protocol is followed. However, occasionally a large fraction of cells is dead. The causes are sometimes unknown, and an error must be suspected. If a second, similar incident occurs by another experimenter soon thereafter, a structured search for a potential error source is indicated. In our department, a cell toxic impurity of one of the cell culture medium ingredients accidently introduced by the manufacturer had caused such an error. Only through swift and systematic investigation we were able to minimize the waste of resources, not only in our laboratories, but also for other customers of the manufacturer of the cell culture medium.

Human errors
In any work environment in which people are planning and executing tasks so called 'human errors' are bound to happen. Main causes include carelessness or moments of distraction during complex tasks. Most often human errors in the biomedical lab present themselves as mix-ups, for instance when a wrong reagent is used with a similar appearance, or samples in identical containers get mixed up or when a container received a wrong label (Figure 1).
Examples for negligence include the failure to close the door of a lab freezer which contains important samples, or the introduction of thermolabile equipment into an autoclave. Despite the fact that many lab records are obtained electronically, human errors can occur when electronic documentation of experiments is not saved, or is accidentally overwritten.

Error management
A number of specific measures can help to reduce the potential for error in biomedical research: Critical reagents which are prone to mix-ups should receive a distinctive color-coding. Critical work steps should be witnessed by another person (known as four-eyeprinciple). Solutions which assist humans by automatizing repetitive workflow or involve alarms are particularly effective, such as freezers with an active alarm system or automatic centralized data backups concurrent with data generation. Most important, however, is that errors are recorded, evaluated and communicated. In the clinical world such error discussion sessions are known as "Morbidity and Mortality" conferences: Medical doctors regularly present cases that involve human errors in order to come up with preventive measures, and to help others making the same mistakes. Another structured approach to error communication and prevention are anonymous reporting tools known as Critical Incidence Reporting Systems (CIRS). Up to now such a stringent error reporting practice only takes place in highly regulated environments, such as healthcare, aviation, or power plants.

LabCIRS -a simple and free error management tool for biomedicine
In our department we have developed, implemented, and tested a free and simple error management tool for biomedicine. Laboratory Critical Incidence Reporting System (LabCIRS) is an adaptation of the clinical CIRS model. Biomedical lab personnel is encouraged to report any laboratory practices, results or situations that could negatively impact safety, animal welfare, longevity of material and devices in addition to observed protocol deviation or any other error of yet unknown causes 9 . While this directive seems logic and reasonable, the reality of error handling in biomedical laboratories today is different.
Commonly many errors go unnoticed or are not communicated at all because currently there is no mandatory error management system for preclinical research laboratories. Errors have a negative connotation and there is the fear for personal liability and humiliation which may hinder biomedical lab personnel to report such incidents. We hope with the introduction and free provision of LabCIRS that this will change, as CIRS has changed the error communication culture and has become a standard in the medical field.
A CIRS system was first described by Flanagan in 1954 10 , by 1978 introduced in anesthesiology 11 , and is today a worldwide established mechanism that improves patient safety in complex, fast-paced and often understaffed clinical settings. What made it so attractive to adapt this system for the preclinical world was the fact that all incidents are recorded anonymously, analyzed, discussed and communicated (Figure 1) a further indication that the system is accepted and that reporting of errors has no ill consequences for employees. We sincerely hope that LabCIRS is adopted by many laboratories and departments of the academic research community and will contribute to the development of a much needed open error culture in the preclinical research laboratory.

Summary and Outlook
As any other scientific field, biomedical research is susceptible to various types of errors.
Compared to the aviation industry, clinical medicine, or radiation safety, most errors in the biomedicine do not have potentially life-threatening consequences. Therefore, governmental oversight in biomedical research is minimal and most commonly restricted to work with genetically-modified organisms, animal welfare, occupational health related safety, and environmental protection. Nevertheless, errors in biomedicine may cause major waste of resources, and potentially harm patients if clinical studies are based on erroneous or faulty results. Despite the existence of protocols and guidelines in biomedical laboratories, most errors simply 'occur', and are not systematically followed up for future prevention. Policies, established procedures, or tools regarding error management are virtually non-existent in this environment. We posit that the biomedical research community, which is currently undergoing a 'reproducibility crisis' to which quality issues may contribute substantially, should develop and implement minimal standards for quality management which includes systematic error management strategies [12][13][14] . Experimental Biomedicine" PloS Biology 2016; in press) A researcher mistook two faintly labeled reagents A and B, which ruined his experiment. "'Reporting"': Entry of the incident into LabCIRS. '"Assessment"': A group of experts (scientists and technicians) reviews the error, and takes preventive action by color labelling of the reagents. "'Feedback"': The errors as well as the measure to prevent it in the future are communicated to the entire laboratory.