The Coralscapes Dataset: Semantic Scene Understanding in Coral Reefs

The Coralscapes dataset is the first general-purpose dense semantic segmentation dataset for coral reefs. Similar in scope and with the same structure as the widely used Cityscapes dataset for urban scene understanding, Coralscapes allows for the benchmarking of semantic segmentation models in a new challenging domain. The Coralscapes dataset spans 2075 images at 1024×2048px resolution gathered from 35 dive sites in 5 countries in the Red Sea, labeled in a consistent and speculation-free manner containing 174k polygons over 39 benthic classes.

This repository provides a collection of scripts and instructions for working with the Coralscapes dataset. It includes the full codebase necessary for training and evaluating models on this dataset, allowing to reproduce the results in the paper. Additionally, it contains scripts and step-by-step guidance on how to use the trained models for inference and how to fine-tune the models to external datasets.

The dataset structure of the Coralscapes dataset follows the structure of the Cityscapes dataset:

{root}/{type}/{split}/{site}/{site}_{seq:0>6}_{frame:0>6}_{type}{ext}

The meaning of the individual elements is:

• root the root folder of the Coralscapes dataset.
• type the type/modality of data, gtFine for fine ground truth, leftImg8bit for left 8-bit images, leftImg8bit_1080p (gtFine_1080p) for the images (ground truth) in 1080p resolution, leftImg8bit_videoframes for the 19 preceding and 10 trailing video frames.
• split the split, i.e. train/val/test. Note that not all kinds of data exist for all splits. Thus, do not be surprised to occasionally find empty folders.
• site ID of the site in which this part of the dataset was recorded.
• seq the sequence number using 6 digits.
• frame the frame number using 6 digits.
• ext .png

File Structure

The files provided in the Zenodo repository are the following:

• coralscapes.7z contains the Coralscapes dataset which includes the 2075 images and corresponding ground truth semantic segmentation masks at 1024x2048px resolution.
• coralscapes_1080p.7z contains the Coralscapes images and masks in their native 1080x1920px resolution.
• model_checkpoints.7z contains the checkpoints of the semantic segmentation models that have been fine-tuned on the Coralscapes dataset. This includes the following models: SegFormer (with a B2 and B5 backbone, trained with and without LoRA), DPT (with a DINOv2-Base and DINOv2-Giant backbone, trained with and without LoRA), a Linear segmentation model with a DINOv2-Base backbone, a UNet++ with a ResNet50 backbone and DeepLabV3+ with a ResNet50 backbone.   
• coralscapes_videoframes.7z contains the the 19 preceding and 10 trailing video frames of each image in the Coralscapes dataset.