Published November 12, 2026 | Version 1.0.0

Stabilo Benchmark: Reproducible Static and Dynamic Benchmarks for Reference-Based UAV Video Stabilization

Authors/Creators

  • 1. EPFL

Description

Stabilo Benchmark provides reproducible static and dynamic benchmarks for reference-based UAV video stabilization. It is the benchmark implementation accompanying the paper "Consensus Capture in Reference-Based UAV Video Stabilization: A Benchmark on Dynamic Urban Scenes", and it evaluates the complete geometric registration pipeline (feature detector, descriptor, matcher, robust estimator, and object-masking strategy) rather than any component in isolation. Reference-based stabilization matters wherever object positions must stay consistent over a long recording, such as trajectories in image coordinates or on a map; the paper's evidence comes from vehicles at urban intersections, but the benchmark applies to any annotated objects that move independently of the camera.

Why another benchmark

Existing feature-matching benchmarks measure correspondence accuracy under synthetic geometric and photometric perturbations, on scenes without independently moving content. They do not address dynamic scenes, in which a coherently moving foreground can dominate geometric consensus. A group of objects moving together, such as vehicles crossing an intersection, is self-consistent, so a homography fitted to those objects rather than to the ground still reports a high inlier count, a high inlier ratio, and a low reprojection error. Ranking pipelines by inlier statistics favors exactly that failure.

This benchmark measures how well the estimated homography aligns the static world, not how self-consistent the fit was. Inlier statistics are still recorded, as diagnostics, never as the score.

What it provides

  • Part I, static benchmark: known synthetic homographies plus photometric degradation on static aerial frames, where ground truth is exact by construction. Extends the campaign of the background paper with learning-based detectors and matchers, object-level metrics, sensor noise, and JPEG compression.
  • Part II, dynamic benchmark: long real hovering-drone recordings over busy intersections. Each sampled frame is registered directly onto the reference frame, never by accumulating adjacent transforms, at an offline and a real-time compute budget.
  • An evaluation methodology for dynamic scenes: masked background-alignment metrics (SSIM, ZNCC, mutual information, gradient-orientation similarity) that need no ground-truth homography, a decomposition of registration error along the dominant motion axis of the annotated objects, and a direct measure of consensus capture: how far an estimate follows the largest group of objects that moved together, rather than the ground.
  • Object-level metrics: intersection over union (IoU) and object center error (OCE) of each annotated box after registration, summarized per frame by mean, standard deviation, minimum, quartiles, and maximum. OCE is the position error a trajectory inherits from registration.
  • A ground-control-point audit: the reference estimator is treated as pseudo-ground truth, and its own error is measured against hand-annotated ground-plane points, giving Part II a stated resolution floor below which differences are not interpreted.
  • Three masking strategies: none, horizontal bounding boxes (HBB), and oriented bounding boxes (OBB), applied before feature detection, with the complete benchmark repeated for each.
  • Per-pipeline hyperparameter tuning on a held-out split, with ranking on the integrated accuracy curve (mAA) rather than on a single arbitrary threshold.
  • Manuscript artifacts generated from saved results: LaTeX tables and PDF figures, with no number typed by hand.

Related Projects

  • Stabilo ⚖️: the registration engine this work benchmarks, a Python library for video and trajectory stabilization using robust homography estimation with user-defined masks.
  • Stabilo-Optimize 🎛️: the predecessor, implementing the static campaign reported in the background paper. It remains frozen as that paper's artifact; this repository supersedes it.
  • Geo-trax 🚀: georeferenced vehicle trajectory extraction from high-altitude drone imagery. Its vehicle detector, run through HBB2OBB, supplies the horizontal bounding boxes of both benchmarks.
  • HBB2OBB 🔄: object detection and SAM-assisted conversion of horizontal to oriented bounding boxes; it writes every label set the benchmarks read.

Data

The benchmark's inputs, Part I's annotated scenes and Part II's cases, are released together with the outputs of every campaign as the paper's accompanying data record, which the bundled fetch command downloads and verifies against pinned checksums; a small quick-start sample, used by no result in the paper, is committed to the repository. The frames come from the KAIST–EPFL drone experiment over Songdo, South Korea, and are annotated with HBB2OBB at the configuration selected on the human-verified reference set of Songdo Vision OBB, which the paper releases separately.

Citation

If you use Stabilo Benchmark or its data, please cite the paper:

@article{fonod2026benchmark,
  author  = {Fonod, Robert and Geroliminis, Nikolas},
  title   = {Consensus Capture in Reference-Based UAV Video Stabilization: A Benchmark on Dynamic Urban Scenes},
  journal = {arXiv preprint arXiv:2611.17190},
  year    = {2026},
  doi     = {10.48550/arXiv.2611.17190}
}

When you use the data record, please also cite it:

@dataset{fonod2026benchmarkdata,
  author    = {Fonod, Robert and Geroliminis, Nikolas},
  title     = {Accompanying data for the paper "Consensus Capture in Reference-Based UAV Video Stabilization: A Benchmark on Dynamic Urban Scenes"},
  year      = {2026},
  version   = {v1},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.23623540}
}

The background paper introduced the stabilization approach, the imagery and the static campaign this work extends:

@article{fonod2025advanced,
  author  = {Fonod, Robert and Cho, Haechan and Yeo, Hwasoo and Geroliminis, Nikolas},
  title   = {Advanced computer vision for extracting georeferenced vehicle trajectories from drone imagery},
  journal = {Transportation Research Part C: Emerging Technologies},
  volume  = {178},
  pages   = {105205},
  year    = {2025},
  doi     = {10.1016/j.trc.2025.105205}
}

To cite this version of the software, please cite its archived release:

@software{fonod2026stabilobenchmark,
  author    = {Fonod, Robert},
  title     = {Stabilo Benchmark: Reproducible Static and Dynamic Benchmarks for Reference-Based UAV Video Stabilization},
  year      = {2026},
  version   = {1.0.0},
  license   = {MIT},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.24168256},
  url       = {https://github.com/rfonod/stabilo-benchmark}
}

Contributions

Community contributions are welcome: open a GitHub Issue or submit a pull request.

License

This project is distributed under the MIT License. See the LICENSE file for more details.

Files

rfonod/stabilo-benchmark-v1.0.0.zip

Files (44.4 MB)

Name Size
md5:31613e9dab9c04a879b6c1541be31535
44.4 MB Preview Download

Additional details

Funding

Board of the Swiss Federal Institutes of Technology
Open Research Data (ORD) Program of the ETH Board
Swiss National Science Foundation
NCCR Automation (phase II) 225155

Dates

Created
2026-11-12
Initial public release