Automatic Outlier Detection in Organ at Risk and Target Delineation: A Modality-Independent Approach for Fast Quality Assurance

Background and Purpose

With the growing availability of AI delineation tools for regions of interest (ROI) (i.e. organ at risk and target delineation), clinical implementations are becoming more common. While these tools have shown reductions in delineation time and increased delineation consistency, they are still lacking reliable error detection. In this study, we developed an outlier detection tool (ODT) and assessed its performance.

Materials and Methods

The ODT characterises ROIs by their geometric properties, relative positions and islands and creates a reference model based on principal component analysis (see Fig.1). For each ROI, the tool returns a value between 0 and 1, with values close to 0 indicating an outlier. A prostate cancer dataset consisting of 25 manually delineated clinical scans from 25 patients was used for training, and 1288 AI-delineated scans from 116 patients were used to evaluate the performance.

Results

The tool took on average 6.5 seconds to analyse individual ROIs. We identified 138 cases with a value below 1x10e-9. The majority of identified outliers resulted from bad AI delineations of the femoral heads in cases with hip prosthetics. The tool also found severe outliers where parts of the target were delineated as bowel and bladder.

Conclusion
The tool is able to identify outliers in AI delineations and offers the possibility to investigate the frequency of errors in AI-based delineations in the future. The tool will be available as open-source.