GET-25: The Global Spatiotemporal ADS-B Kinematic Consistency Benchmark

Data License: Creative Commons Attribution 4.0 International (CC BY 4.0)

Code License: MIT

Overview

Welcome to the GET-25 (Global Envelope Thresholding - 2025) dataset. This repository contains the complete open-science codebase, baseline parameters, and extracted spatial datasets for the 2025 global ADS-B kinematic consistency study.

The acronym GET-25 defines the core methodological framework and temporal scope used to generate this data:

• GET (Global Envelope Thresholding): The novel architectural optimization that entirely removes pre-filter survivorship bias, allowing the pipeline to ingest raw, malformed rows to achieve total system observability.

• 25 (2025): The calendar year of the 246.7 billion OpenSky Network ADS-B messages processed to establish this baseline.

Utilizing the OpenSky Network Trino database, this data engineering pipeline establishes the first global, traffic-normalized baseline for surveillance self-consistency across ~77.5 million observed flight hours. By implementing a conservative Maximum Hourly Envelope Threshold and deliberately relaxing standard source filters, the pipeline achieves total system observability. This approach successfully captured over 3.8 million "ghost records" (severe kinematic anomalies lacking standard avionics telemetry) that traditional data pipelines systematically discard.

Scope Note: This dataset evaluates the internal statistical consistency of broadcast state vectors (e.g., extreme positional jumps, sensor divergences) as received by the ground network (the "Observer's Reality"). It strictly avoids causal attribution (e.g., hardware faults, GNSS spoofing, or ANSP performance).

Primary Use Cases

• Machine Learning / Cybersecurity: Serves as the empirical "null hypothesis" baseline for natural system noise. ML anomaly classifiers (e.g., GNSS spoofing detectors) can benchmark against this dataset to prevent falsely classifying standard network latency or oceanic coverage gaps as cyber-attacks.

• Advanced Air Mobility (AAM) & UTM: Provides critical baseline data on foundational 1090 MHz spectrum stability, necessary for developing probabilistic risk models for automated low-altitude and eVTOL operations.

• Avionics Safety (ACAS X): Maps the spatial frequency of "Ghost Records" that threaten Markov Decision Process (MDP) state estimations in automated collision avoidance systems.

Quick Start & Reproducibility

To replicate the GET-25 extraction pipeline on your own local datasets or OpenSky access:

1. Install dependencies: pip install -r requirements.txt

2. Execute the pipeline sequentially via the scripts/pipeline_stages/ directory (Stage 1 through Stage 5b).

3. Run the validation suite in scripts/validation_tools/ to ensure spatial and temporal parity with the published baseline.

Zenodo File Manifest

Due to Zenodo's flat file structure, this repository has been packaged into modular .zip archives.

1. Root Meta-Files: Contains the dataset README.md, the LICENSE.txt, and the requirements.txt.

2. processed_events_2025_XX.zip (12 Files): Contains the final extracted anomaly dataset (Numerator). Extract to data/processed_kinematic_events/ to create daily .parquet files.

3. exposure_grids_2025_XX.zip (12 Files): Contains daily 0.1-degree spatial grids of baseline flight exposure. Extract to data/reference_and_baselines/exposure_grids/.

4. reference_and_baselines.zip: Contains the denominator and geographical mapping files:

   • fir_exposure_reference.csv: Total global flight hours aggregated per FIR.

   • global_kinematic_thresholds.csv: The absolute P95/P99 empirical constants.

   • sensitivity_analysis_V2.5-2025_hourly.csv: The hourly volumetric baseline.

5. scripts.zip: Contains the exact Python architecture used to query the 246-billion-row cluster and model the data globally.

6. results.zip: Contains descriptive validation outputs and the final statistical tables (Regional Incidence Rate Ratios and Traffic-Normalized FIR anomaly rates).

Directory Structure

ADSB_2025_Release/
├── README.md
├── LICENSE.txt
├── requirements.txt
├── data/
│   ├── processed_kinematic_events/
│   │   └── [12 monthly .zip archives containing daily .parquet files]
│   └── reference_and_baselines/
│       ├── fir_exposure_reference.csv
│       ├── global_kinematic_thresholds.csv
│       ├── sensitivity_analysis_V2.5-2025_hourly.csv
│       └── exposure_grids/
│           └── [12 monthly .zip archives containing daily .parquet grids]
├── results/
│   ├── extraction_statistics_2025.csv
│   ├── ghost_records_summary_2025.csv
│   ├── table2_top_bottom_firs.csv
│   └── table5_IRR_results.csv
└── scripts/
    ├── pipeline_stages/
    │   ├── stage1_sensitivity_analysis.py
    │   ├── stage1b_exposure_baseline.py
    │   ├── stage2_threshold_consolidation.py
    │   ├── stage3_event_extraction.py
    │   ├── stage4_spatial_enrichment.py
    │   ├── stage5_glmm_irr_analysis.py
    │   └── stage5b_fir_level_analysis.py
    └── validation_tools/
        ├── validate_annual_temporal_stats.py
        ├── validate_ghost_records.py
        ├── validate_hourly_integrity.py
        └── validate_spring_volatility_stats.py

Methodological Nuances & Limitations

Analysts utilizing this dataset must account for the following structural realities of the ADS-B network, as documented in the associated manuscript:

• The "Data Gap" Confounding: Oceanic and remote FIRs (e.g., NAT, PACI) inherently experience high latency. Kinematic jumps triggered by these coverage gaps are retained in the dataset as genuine degradations of systemic observability.

• Receiver Heterogeneity: The exposure offset perfectly corrects for traffic volume disparities but does not correct for crowdsourced receiver density. High-density regions (like Europe) may exhibit "sensor divergence" artifacts due to multi-receiver triangulation conflicts.

• Multi-Trigger Events: Traffic-normalized rates in highly volatile regions can mathematically exceed 3,600 events per flight hour. This occurs because a single anomalous message can simultaneously breach multiple independent kinematic thresholds (e.g., Speed, Altitude, and Heading).

• Antimeridian Continuity: The Haversine extraction logic explicitly accounts for the ±180° antimeridian wrap-around in the Pacific, preventing standard geometric processing engines from falsely flagging standard trans-Pacific flights as "teleportation" anomalies.

How to Cite

If you utilize this dataset or empirical baseline in your research, please cite the accompanying manuscript:

Pik, E., Garcia, J. S. D., Smith, T. A. (2026). The Invisible Infrastructure: A Global Baseline for ADS-B Kinematic Integrity and the Ghost Record Phenomenon. [Journal Name/DOI pending].

Dataset DOI: 10.5281/zenodo.18904147