AI Nuclei Detection

This repository provides a complete workflow and software toolkit for automated nuclei detection on immunohistochemically stained images (specifially designed to work with complex ECMO membrane lung fiber mat images). The project implements a Mask R‑CNN–based instance segmentation model built with PyTorch (v2.2.0) and Detectron2 framework, along with training utilities and a ready‑to‑use Windows application for end‑users.

The training and desktop application rely heavily on the following packages: 

• Computer Vision Annotation Tool (CVAT)
• Detectron2 by Facebook AI Research
• Slicing Aided Hyper Inference (sahi) by Open Business Software Solutions
• PyOneDark Modern GUI

Project Overview

Accurate nuclei detection and shape analysis is a crucial step in analysing cellular deposits on membrane fibers to better understand why clotting occurs in ECMO.

This repository delivers an end‑to‑end machine learning solution tailored to this domain:

• A custom desktop application optimized for nuclei identification, clustering and shape analysis.
• Tools to pre-process image data
• Evaluation and comparison against established open‑source tools such as Cellpose and StarDist.

Repository Structure

training/ – Training Utilities

Includes scripts and helper modules for:

• Data pre-processing of COCO formated annotations exported from Computer Vision Annotation Tool (CVAT) (01_balance_coco_data.py, 02_split_coco_dataset_into_patches.py)
• Data used for training: training/TRAINDATA/sliced_coco
• Train script (03_train_model.py)
• Model evaluation (04_evaluate_model.py)
• Visualization of ground truth and predictions (05_inference.py)

comparison/ – Comparison against available nuclei detection software

Contains:

• Scripts to run inference using Cellpose (comparison/model_segm_comparison/cellpose/run_cellpose_on_images.py) and StarDist (comparison/model_segm_comparison/stardist/run_stardist_on_images.py)
• Trained model checkpoints (in mask_rcnn_configurations/mask_rcnn_configurations)
• Script to compare nuclei count and relative area of trained Mask R-CNN configurations with Cellpose and Stardist (model_segm_comparison.py)

app/ – Python‑Based Windows Application

A standalone Windows desktop application that allows end users to:

• Load tiff images
• Run nuclei detection using the trained Mask R‑CNN model
• Detect nulcei clusters
• Visualize and export segmentation results
• Generate summary statistics to PDF file

Installation

Python IDE

To run the desktop application /app/main.py in your Python IDE, downalod Python (v3.10.19), create a virtual environment and install all required packages using 

#Create a virtual environment
python -m venv env

#Activate the environment
env\Scripts\activate

#Install required packages
pip install -r requirements.txt

Windows Desktop Application

See /Windows_Portable and /Windows_Installer for installing the bundled Windows executable on your system.

You can use /Windows_Portable for a portable installation and /Windows_Installer for installation in the user's application folder (AppData) or another user defined path.