MMoRWaD - The Multi Modal Recyclable Waste Management Dataset

Description

Abstract

Automated classification of municipal solid waste is challenging due to large variability in object appearance, deformation, contamination, and disposal behavior. To address this, we present a multimodal dataset collected using a custom reverse-vending-machine station equipped with an Intel RealSense D405 RGB-D sensor and a ReSpeaker Mic Array v2.0, capturing co-recorded RGB images, depth data, video, and impact audio from recyclable household waste. The dataset contains 3,300 multimodal samples from 721 distinct items, collected in Thessaloniki, Greece, between May 2024 and January 2025, and is fully annotated with recycling-relevant attributes, including material type, color, and the presence of caps, labels, contamination, and deformation. COCO annotations are also provided to support object detection and segmentation tasks. This dataset is intended to support the development and evaluation of robust machine-learning models for automated waste classification.

Table of contents

The directory structure is organized by modality into six main components: RGB images, depth images, RGB videos, depth videos, audio recordings, and metadata files. Each modality is stored in a dedicated directory following a consistent naming convention.

In addition to the raw data, three annotation files are provided:
• a CSV file containing all classification attributes (material, color, cap, label, contamination, deformation),
• a COCO-format JSON file with instance segmentation masks, bounding boxes, categories, and item names,
• a supplementary JSONL file recording the number of captures per item along with the corresponding item name, ensuring consistent integration of new samples into the dataset.

All RGB and depth images are stored in PNG format at a resolution of 848 × 480 pixels. RGB and depth videos share the same resolution, are recorded at 30 fps, have an average duration of approximately 2 seconds, and are stored in AVI format. Audio recordings are provided in WAV format, captured as mono signals at 48 kHz and 16-bit resolution.

Technical info

Naming Convention

A structured file-naming scheme encodes the main properties of each waste item directly in the filename, enabling fast identification and filtering without inspecting file contents.

Filename format:

<ID><data_type>_MxxHxxTxLxCxDx

Where:

• ID: incremental numerical identifier

• data_type: data modality (ic, id, a, vc, vd)

• Mxx: material code

• Hxx: hue/color code

• Tx: cap presence

• Lx: label presence

• Cx: contamination

• Dx: deformation

ic = image color, id = image depth, a = audio, vc = video color, vd = video depth.

Material Codes (Mxx)

The material codes follow standard recycling identification symbols commonly used on consumer packaging. These codes allow material type to be inferred directly from the filename and align with widely adopted sorting practices.

Hue Codes (Hxx)

Binary Attribute Codes

Dataset Distributions

This section provides a concise visual overview of the dataset composition, focusing on material balance and key physical attributes relevant to recycling scenarios.

Material Distribution

type is summarized below.

Plastic

• Polyethylene Terephthalate (PET): 1,146 samples (34.8%)

• High Density Polyethylene (HDPE): 488 samples (14.8%)

• Polypropylene (PP): 432 samples (13.1%)

• Other Plastics: 103 samples (3.1%)

• Low Density Polyethylene (LDPE): 35 samples (1.1%)

• Polystyrene (PS): 14 samples (0.4%)

• Polyvinyl Chloride (PVC): 5 samples (0.2%)

Paper and Cardboard

• Paperboard (PAP): 375 samples (11.4%)

• Cardboard (PAP): 61 samples (1.8%)

• Paper (PAP): 38 samples (1.2%)

Metal

• Aluminum (ALU): 167 samples (5.1%)

• Steel (FE): 77 samples (2.3%)

Glass

• Clear Glass (GL): 109 samples (3.3%)

• Brown Glass (GL): 36 samples (1.1%)

• Green Glass (GL): 10 samples (0.3%)

Composite Materials

• Paper/Cardboard + Plastic + Aluminum (C/PAP): 96 samples (2.9%)

• Paper/Cardboard + Plastic (C/PAP): 76 samples (2.3%)

• Plastic + Aluminum (C/PE): 32 samples (1.0%)

Color Distribution

Color information is included to capture variations relevant to material identification and recycling. Plastic items exhibit the widest color diversity, ranging from transparent PET to highly saturated and multicolored packaging. Paper items are predominantly annotated as “other” due to printed surfaces, with brown used mainly for cardboard. Metal items are mostly gray or multicolored, while glass colors are directly tied to material type and limited to clear, brown, and green.

Binary Attribute Distribution

Binary attributes include cap presence, label presence, contamination, and deformation. Plastic and composite items show the highest variability across these attributes, particularly due to caps and removable labels. Paper and metal items rarely include caps or separate labels, as information is typically printed directly on the surface. Glass items frequently include caps and labels but show no deformation, while contamination is generally low across all materials.

Dataset Limitations

• Multiple samples come from the same object under different conditions. Dataset splits should be done at the object level (using the CSV) for evaluation requiring independence.

• Data were collected in a controlled environment with fixed lighting and background. While this ensures consistency and reliable annotations, it may limit generalization to unconstrained real-world settings.

• Contamination, when needed, was reproduced with a restricted set of residues. Liquid residue was recorded only in containers whose closure was in place, in order to protect the acquisition equipment, so contaminated samples of opened liquid containers are absent. Dry residue was recorded in packaging without a closure as well, and the contaminated class therefore includes uncapped cartons, boxes and bags. Contamination is not independent of closure or material: it is recorded in 42.7% of capped samples against 13.2% of uncapped ones, and 251 of the 254 contaminated samples without a cap come from packaging that has no closure anywhere in the dataset. Labels printed directly on the surface were annotated as absent, as they do not require separation before recycling.

• A band along the left edge of the depth frames falls outside the region observed by both imagers of the stereo pair, so no disparity is available there and the hole-filling filter populates it with values propagated from the frame border. The band is present in all 3,300 depth images. Its incidence is highest at the frame edge and decays with distance from it, dropping below 1% of pixels beyond column 52 of 848, and it accounts on average for 2.7% of the pixels of a depth frame. The leftmost annotated object boundary across the dataset lies at column 137, so in the still depth images the band does not overlap the item and can be removed by cropping. In the depth videos the item crosses the field of view as it falls and may enter the band during the first frames of a capture, in which case the part of the item inside the band carries no valid depth. The depth streams were recorded under the factory calibration of the device without a standardized accuracy evaluation, so the released values should be treated as relative rather than certified metric measurements.

• Visually similar items may belong to different material categories. This is a problem inherent to recycling.

• The dataset reflects packaging waste collected in Thessaloniki, Greece. Packaging placed on the European market is subject to harmonised requirements, so the packaging types and structural characteristics recur across Member States, but regional differences in products, brands and packaging design may affect model performance on waste streams from other geographical contexts.

• Impact acoustics depend on the drop height, the orientation of the item and the location of the impact. The manual release procedure fixes only the release opening and the impact surface, so the recorded audio carries variability from the drop in addition to variability from the item, and acoustic differences between two captures of the same item are not attributable to the item alone. 

• Residue was reproduced with a small set of substitutes, water for liquid products and oats or beans for solid ones, chosen for consistency with the original content. A model trained on the impact audio may therefore learn the acoustic signature of these particular substances rather than a general notion of contamination. The choice of substitute follows the product type, so residue type is correlated with packaging type and therefore with material.

Hardware and Structural Components of the Acquisition Station

The acquisition station is composed of sensing and computing hardware together with an enclosure and supporting structural materials. The listed components and specifications correspond to the configuration used during data acquisition.

Technical info

Official Train/Validation/Test Splits

This record includes official train/validation/test splits of the MMoRWaD dataset, defined at the physical-object level to prevent data leakage between splits.

Why object-level splits?

Each physical waste item was recorded multiple times (3,300 samples in total, captured from 721 distinct items). Splitting at the sample level would place recordings of the same physical object in both training and test sets, inflating performance estimates. The official splits therefore assign every sample of a given physical item to exactly one split: no object appears in more than one split, and no sample-level leakage is possible.

Split methodology

1. Object grouping. Samples are grouped by the commercial name of the item, as recorded in the acquisition system's item-count log (721 distinct items). Each group corresponds to one physical object.
2. Primary material assignment. Each object is assigned a primary material — the fine-grained material class of the majority of its samples (ties broken by the lowest sample ID). This handles the few objects whose samples span more than one material class.
3. Brand grouping. Objects are further grouped into commercial brands by their commercial name (335 brands), so that identical packaging designs or commercial graphics cannot appear in more than one split.
4. Brand-disjoint split. Objects are assigned to train/validation/test by solving a mixed-integer linear program that minimizes the per-material deviation from the target 80/10/10 ratio, subject to the constraint that no brand spans more than one split. The split is deterministic and fully reproducible.
5. Minimum-coverage rule. Material classes with at least three objects are guaranteed at least one validation and one test object.
6. Verification. The split was verified programmatically: all 3,300 samples are covered, object and sample files are consistent, no object spans more than one split, and no brand spans more than one split.

Files

Split properties

Object distribution by broad material category

Reproducibility

The splits are generated deterministically by the accompanying script (make_splits.py), which reads the dataset table and the item-count log, solves the brand-disjoint assignment, and regenerates all four files. No random seed is involved; the same input files always produce the same split.

Notes

Notes