There is a newer version of the record available.

Published April 22, 2026 | Version 1.0

Degradation of ADS-B Surveillance Integrity: Temporal Clustering and Anomaly Persistence

Authors/Creators

  • 1. Mevocopter Aerospace

Description

Data License: Creative Commons Attribution 4.0 International (CC BY 4.0)

Code License: MIT

Overview

Welcome to the replication dataset for the study "Degradation of ADS-B Surveillance Continuity: Temporal Clustering and Anomaly Persistence". This repository contains the complete open-science codebase, raw extraction shards, pre-computed survival matrices, and statistical artifacts required to fully reproduce the longitudinal analysis of 2025 global ADS-B tracking stability.

Extracting from a gross volume of 246.7 billion OpenSky Network ADS-B messages logged across the 2025 calendar year, this study analyzes 229.0 billion filtered, airborne state vectors. The customized, out-of-core computational pipeline executes scrambled-temporal permutation testing ($\mathcal{W}=\{3,5,7,10,15\}$), time-to-recurrence Kaplan-Meier survival modeling, and exposure-adjusted Cox proportional hazards regression to quantify longitudinal surveillance continuity across 233,859 unique aircraft trajectories.

Quick Start & Reproducibility

To replicate the temporal clustering and survival modeling pipeline locally:

  • Install dependencies: pip install -r requirements.txt

  • Execute Stage 1: Run CODE/stage1_trino_extraction.py on the OpenSky Trino cluster to extract sequential data (Note: Requires OpenSky credentials).

  • Execute Stage 2: Run CODE/stage2_data_generation.py to execute the Numba JIT-compiled permutation tests and build the survival cohorts.

  • Execute Stage 3: Run CODE/stage3_figure_generation.py to ingest the Stage 2 .parquet files and rapidly re-render all manuscript PDFs and CSV tables.

Directory Structure

.
├── README.md
├── requirements.txt
├── CODE/
│   ├── config.py
│   ├── extract_manuscript_data.py
│   ├── stage1_trino_extraction.py
│   ├── stage2_data_generation.py
│   └── stage3_figure_generation.py
├── DATA/
│   ├── H2_enhanced_results.parquet
│   └── extractions/
│       ├── Dataset_2A_aggregated_2025XXXX.PARQUET.zip
│       ├── Dataset_2B_daily_spans_2025XXXX.PARQUET.zip
│       ├── Dataset_2B_ledger_2025XXXX.PARQUET.zip
│       └── Q1_diagnostic_stats_2025XXXX.CSV.zip
├── LOGS/
│   ├── stage1_trino_extraction.log
│   ├── stage2_data_generation.log
│   └── stage3_figure_generation.log
└── RESULTS/
    ├── data_permutation_validation.csv
    ├── df_gaps_raw.parquet
    ├── df_ledger.parquet
    ├── fig_censoring_sensitivity_data.csv
    ├── fig_continental_sensitivity_data.csv
    ├── fig_exposure_stratified_survival_data.csv
    ├── fig_stratified_survival_data.csv
    ├── gap_stats.parquet
    ├── manuscript_fill_values.json
    ├── Master_Survival_Cohort.parquet
    ├── model_cox_coefficients.csv
    ├── model_schoenfeld_residuals.txt
    ├── tab_block_sensitivity.csv
    ├── tab_censoring_demographics.csv
    ├── tab_per_aircraft_clustering.csv
    └── figures/
        ├── Figure_1_pipeline_architecture.pdf
        ├── Figure_2_stratified_recurrence.pdf
        ├── Figure_3_cox_exposure_stratified.pdf
        ├── Figure_S1_censoring_sensitivity.pdf
        ├── Figure_S2_dropout_sensitivity.pdf
        ├── Figure_S3_penalizer_sensitivity.pdf
        └── Figure_S4_continental_sensitivity.pdf

File Manifest & Data Dictionary

1. Root Files

  • README.md: This documentation file.

  • requirements.txt: Standardized list of Python dependencies (e.g., numba, lifelines, pyarrow) to ensure environment reproducibility.

2. CODE/ (The Computational Pipeline)

  • config.py: The central configuration module. Contains the exact kinematic anomaly thresholds (e.g., speed $> 6670.3$ m/s), memory-guard limits, and temporal window size parameters used universally across all pipeline stages.

  • extract_manuscript_data.py: A parsing utility that reads the logs, CSVs, and Parquet files to dynamically extract and format the macro-statistics (like total flight hours and fleet sizes) needed for the manuscript text.

  • stage1_trino_extraction.py: The distributed database extraction script. Executes native SQL window functions on the OpenSky Trino cluster to download and pair daily trajectory trajectories.

  • stage2_data_generation.py: The core data science engine. Runs the out-of-core scrambled-temporal permutation tests (via Numba JIT compilation) and aggregates raw trajectories into discrete monthly survival datasets.

  • stage3_figure_generation.py: The visualization engine. Ingests Stage 2 data to compute the Cox Proportional Hazards models, log-rank tests, and renders the publication-ready PDF figures and CSV tables.

3. DATA/ (Inputs and Intermediates)

  • H2_enhanced_results.parquet: The pre-computed master results matrix from the Stage 2 permutation tests, containing Z-statistics, O/E ratios, and FDR-corrected p-values for all evaluated window scales.

  • extractions/Dataset_2A_aggregated_*.PARQUET.zip: Raw binary anomaly vectors used for independent multi-scale testing.

  • extractions/Dataset_2B_daily_spans_*.PARQUET.zip: Daily observation boundaries used to track network dropouts.

  • extractions/Dataset_2B_ledger_*.PARQUET.zip: The severity-classified anomaly ledger.

  • extractions/Q1_diagnostic_stats_*.CSV.zip: Daily data quality reports (e.g., duplicate message rates).

4. LOGS/

  • stage1_trino_extraction.log: Extraction timing and data volume logs.

  • stage2_data_generation.log: Permutation tracking and cohort construction logs.

  • stage3_figure_generation.log: Modeling outputs, including the in-sample Cox Model C-Index.

5. RESULTS/ (Final Datasets, Tables, and Models)

  • Master_Survival_Cohort.parquet: The core discrete life-table dataset. It anchors $T_0$ to individual observation inception to prevent immortal time bias and provides the basis for all Kaplan-Meier and Cox models.

  • df_ledger.parquet: Reconciled monthly ledger mapping raw anomalies to their specific physical Severity Classes.

  • df_gaps_raw.parquet: Consolidated raw daily observation spans.

  • gap_stats.parquet: Processed inter-observation gaps used to enforce 30-day network dropout right-censoring.

  • data_permutation_validation.csv: Results of the empirical validation subset confirming high classification concordance between $N=200$ and $N=10,000$ permutations.

  • model_cox_coefficients.csv: Complete Cox Proportional Hazards regression outputs ($\beta$ coefficients, hazard ratios, $p$-values).

  • model_schoenfeld_residuals.txt: Outputs from the scaled Schoenfeld residuals test evaluating the proportional hazards assumption.

  • manuscript_fill_values.json: An auto-generated JSON key-value store containing all dynamic scalars referenced directly in the manuscript body.

6. RESULTS/ (Manuscript Tables & Curve Coordinates)

  • tab_per_aircraft_clustering.csv: Per-aircraft temporal clustering statistics across all window scales.

  • tab_block_sensitivity.csv: Comparison of significance rates evaluating the uniform-null versus the stricter block-preserving null model.

  • tab_censoring_demographics.csv: Demographic profiles of the resulting survival cohorts by censoring status.

  • fig_stratified_survival_data.csv: Coordinate data for Figure 2 (Severity Class stratification).

  • fig_exposure_stratified_survival_data.csv: Coordinate data for Figure 3 (Flight hours tertile stratification).

  • fig_censoring_sensitivity_data.csv: Coordinate data for Figure S1 (Complete cases vs baseline).

  • fig_continental_sensitivity_data.csv: Coordinate data for Figure S4 (Continental-only subset).

7. RESULTS/figures/ (Publication Graphics)

  • Figure_1_pipeline_architecture.pdf: Methodological flowchart.

  • Figure_2_stratified_recurrence.pdf: Time-to-first anomaly survival functions by physical Error Mode.

  • Figure_3_cox_exposure_stratified.pdf: Exposure-adjusted survival hazard stratifications.

  • Figure_S1_censoring_sensitivity.pdf: Sensitivity impact of network dropouts vs administrative censoring.

  • Figure_S2_dropout_sensitivity.pdf: Stability check across 15, 30, 45, and 60-day network dropout thresholds.

  • Figure_S3_penalizer_sensitivity.pdf: Model coefficient stability versus L2 Ridge penalizer variance.

  • Figure_S4_continental_sensitivity.pdf: Sensitivity control isolating oceanic line-of-sight dropouts.

How to Cite

If you utilize this dataset, pipeline architecture, or empirical baseline in your research, please cite the accompanying manuscript:

Pik, E. (2026). Degradation of ADS-B Surveillance Continuity: Temporal Clustering and Anomaly Persistence. Aerospace Science and Technology (Under Review).

Dataset DOI: 10.5281/zenodo.19698275

Notes (English)

Funding & Acknowledgements

Funding: This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Acknowledgements: The author explicitly thanks the OpenSky Network, its founders, and the global network of receiver operators. The crowd-sourced collection and maintenance of this massive terrestrial surveillance database made this planetary-scale longitudinal analysis possible.

Files

README.md

Files (3.0 GB)

Name Size
md5:1d7b186a6f2960edee2646ede15c3a92
46.8 kB Preview Download
md5:b628844b3a35e855392fa3eca60b32fc
2.5 GB Preview Download
md5:7508edbfb48803581c6d255021663583
129.5 kB Preview Download
md5:d008185a94318951f819e413c2dd44eb
10.0 kB Preview Download
md5:3f9ce1a650f67c6772ffb8ede1983ad7
360 Bytes Preview Download
md5:ebde8e03506ba16c36df97063eca0beb
437.9 MB Preview Download

Additional details

Related works

Is supplement to
Dataset: 10.5281/zenodo.18904146 (DOI)