Degradation of ADS-B Surveillance Integrity: Temporal Clustering and Anomaly Persistence
Description
Data License: Creative Commons Attribution 4.0 International (CC BY 4.0)
Code License: MIT
Overview
Welcome to the replication dataset for the study "Degradation of ADS-B Surveillance Continuity: Temporal Clustering and Anomaly Persistence". This repository contains the complete open-science codebase, raw extraction shards, pre-computed survival matrices, and statistical artifacts required to fully reproduce the longitudinal analysis of 2025 global ADS-B tracking stability.
Extracting from a gross volume of 246.7 billion OpenSky Network ADS-B messages logged across the 2025 calendar year, this study analyzes 229.0 billion filtered, airborne state vectors. The customized, out-of-core computational pipeline executes scrambled-temporal permutation testing ($\mathcal{W}=\{3,5,7,10,15\}$), time-to-recurrence Kaplan-Meier survival modeling, and exposure-adjusted Cox proportional hazards regression to quantify longitudinal surveillance continuity across 233,859 unique aircraft trajectories.
Quick Start & Reproducibility
To replicate the temporal clustering and survival modeling pipeline locally:
-
Install dependencies:
pip install -r requirements.txt -
Execute Stage 1: Run
CODE/stage1_trino_extraction.pyon the OpenSky Trino cluster to extract sequential data (Note: Requires OpenSky credentials). -
Execute Stage 2: Run
CODE/stage2_data_generation.pyto execute the Numba JIT-compiled permutation tests and build the survival cohorts. -
Execute Stage 3: Run
CODE/stage3_figure_generation.pyto ingest the Stage 2.parquetfiles and rapidly re-render all manuscript PDFs and CSV tables.
Directory Structure
.
├── README.md
├── requirements.txt
├── CODE/
│ ├── config.py
│ ├── extract_manuscript_data.py
│ ├── stage1_trino_extraction.py
│ ├── stage2_data_generation.py
│ └── stage3_figure_generation.py
├── DATA/
│ ├── H2_enhanced_results.parquet
│ └── extractions/
│ ├── Dataset_2A_aggregated_2025XXXX.PARQUET.zip
│ ├── Dataset_2B_daily_spans_2025XXXX.PARQUET.zip
│ ├── Dataset_2B_ledger_2025XXXX.PARQUET.zip
│ └── Q1_diagnostic_stats_2025XXXX.CSV.zip
├── LOGS/
│ ├── stage1_trino_extraction.log
│ ├── stage2_data_generation.log
│ └── stage3_figure_generation.log
└── RESULTS/
├── data_permutation_validation.csv
├── df_gaps_raw.parquet
├── df_ledger.parquet
├── fig_censoring_sensitivity_data.csv
├── fig_continental_sensitivity_data.csv
├── fig_exposure_stratified_survival_data.csv
├── fig_stratified_survival_data.csv
├── gap_stats.parquet
├── manuscript_fill_values.json
├── Master_Survival_Cohort.parquet
├── model_cox_coefficients.csv
├── model_schoenfeld_residuals.txt
├── tab_block_sensitivity.csv
├── tab_censoring_demographics.csv
├── tab_per_aircraft_clustering.csv
└── figures/
├── Figure_1_pipeline_architecture.pdf
├── Figure_2_stratified_recurrence.pdf
├── Figure_3_cox_exposure_stratified.pdf
├── Figure_S1_censoring_sensitivity.pdf
├── Figure_S2_dropout_sensitivity.pdf
├── Figure_S3_penalizer_sensitivity.pdf
└── Figure_S4_continental_sensitivity.pdf
File Manifest & Data Dictionary
1. Root Files
-
README.md: This documentation file. -
requirements.txt: Standardized list of Python dependencies (e.g.,numba,lifelines,pyarrow) to ensure environment reproducibility.
2. CODE/ (The Computational Pipeline)
-
config.py: The central configuration module. Contains the exact kinematic anomaly thresholds (e.g., speed $> 6670.3$ m/s), memory-guard limits, and temporal window size parameters used universally across all pipeline stages. -
extract_manuscript_data.py: A parsing utility that reads the logs, CSVs, and Parquet files to dynamically extract and format the macro-statistics (like total flight hours and fleet sizes) needed for the manuscript text. -
stage1_trino_extraction.py: The distributed database extraction script. Executes native SQL window functions on the OpenSky Trino cluster to download and pair daily trajectory trajectories. -
stage2_data_generation.py: The core data science engine. Runs the out-of-core scrambled-temporal permutation tests (via Numba JIT compilation) and aggregates raw trajectories into discrete monthly survival datasets. -
stage3_figure_generation.py: The visualization engine. Ingests Stage 2 data to compute the Cox Proportional Hazards models, log-rank tests, and renders the publication-ready PDF figures and CSV tables.
3. DATA/ (Inputs and Intermediates)
-
H2_enhanced_results.parquet: The pre-computed master results matrix from the Stage 2 permutation tests, containing Z-statistics, O/E ratios, and FDR-corrected p-values for all evaluated window scales. -
extractions/Dataset_2A_aggregated_*.PARQUET.zip: Raw binary anomaly vectors used for independent multi-scale testing. -
extractions/Dataset_2B_daily_spans_*.PARQUET.zip: Daily observation boundaries used to track network dropouts. -
extractions/Dataset_2B_ledger_*.PARQUET.zip: The severity-classified anomaly ledger. -
extractions/Q1_diagnostic_stats_*.CSV.zip: Daily data quality reports (e.g., duplicate message rates).
4. LOGS/
-
stage1_trino_extraction.log: Extraction timing and data volume logs. -
stage2_data_generation.log: Permutation tracking and cohort construction logs. -
stage3_figure_generation.log: Modeling outputs, including the in-sample Cox Model C-Index.
5. RESULTS/ (Final Datasets, Tables, and Models)
-
Master_Survival_Cohort.parquet: The core discrete life-table dataset. It anchors $T_0$ to individual observation inception to prevent immortal time bias and provides the basis for all Kaplan-Meier and Cox models. -
df_ledger.parquet: Reconciled monthly ledger mapping raw anomalies to their specific physical Severity Classes. -
df_gaps_raw.parquet: Consolidated raw daily observation spans. -
gap_stats.parquet: Processed inter-observation gaps used to enforce 30-day network dropout right-censoring. -
data_permutation_validation.csv: Results of the empirical validation subset confirming high classification concordance between $N=200$ and $N=10,000$ permutations. -
model_cox_coefficients.csv: Complete Cox Proportional Hazards regression outputs ($\beta$ coefficients, hazard ratios, $p$-values). -
model_schoenfeld_residuals.txt: Outputs from the scaled Schoenfeld residuals test evaluating the proportional hazards assumption. -
manuscript_fill_values.json: An auto-generated JSON key-value store containing all dynamic scalars referenced directly in the manuscript body.
6. RESULTS/ (Manuscript Tables & Curve Coordinates)
-
tab_per_aircraft_clustering.csv: Per-aircraft temporal clustering statistics across all window scales. -
tab_block_sensitivity.csv: Comparison of significance rates evaluating the uniform-null versus the stricter block-preserving null model. -
tab_censoring_demographics.csv: Demographic profiles of the resulting survival cohorts by censoring status. -
fig_stratified_survival_data.csv: Coordinate data for Figure 2 (Severity Class stratification). -
fig_exposure_stratified_survival_data.csv: Coordinate data for Figure 3 (Flight hours tertile stratification). -
fig_censoring_sensitivity_data.csv: Coordinate data for Figure S1 (Complete cases vs baseline). -
fig_continental_sensitivity_data.csv: Coordinate data for Figure S4 (Continental-only subset).
7. RESULTS/figures/ (Publication Graphics)
-
Figure_1_pipeline_architecture.pdf: Methodological flowchart. -
Figure_2_stratified_recurrence.pdf: Time-to-first anomaly survival functions by physical Error Mode. -
Figure_3_cox_exposure_stratified.pdf: Exposure-adjusted survival hazard stratifications. -
Figure_S1_censoring_sensitivity.pdf: Sensitivity impact of network dropouts vs administrative censoring. -
Figure_S2_dropout_sensitivity.pdf: Stability check across 15, 30, 45, and 60-day network dropout thresholds. -
Figure_S3_penalizer_sensitivity.pdf: Model coefficient stability versus L2 Ridge penalizer variance. -
Figure_S4_continental_sensitivity.pdf: Sensitivity control isolating oceanic line-of-sight dropouts.
How to Cite
If you utilize this dataset, pipeline architecture, or empirical baseline in your research, please cite the accompanying manuscript:
Pik, E. (2026). Degradation of ADS-B Surveillance Continuity: Temporal Clustering and Anomaly Persistence. Aerospace Science and Technology (Under Review).
Dataset DOI: 10.5281/zenodo.19698275
Notes (English)
Files
README.md
Files
(3.0 GB)
| Name | Size | |
|---|---|---|
|
md5:1d7b186a6f2960edee2646ede15c3a92
|
46.8 kB | Preview Download |
|
md5:b628844b3a35e855392fa3eca60b32fc
|
2.5 GB | Preview Download |
|
md5:7508edbfb48803581c6d255021663583
|
129.5 kB | Preview Download |
|
md5:d008185a94318951f819e413c2dd44eb
|
10.0 kB | Preview Download |
|
md5:3f9ce1a650f67c6772ffb8ede1983ad7
|
360 Bytes | Preview Download |
|
md5:ebde8e03506ba16c36df97063eca0beb
|
437.9 MB | Preview Download |
Additional details
Related works
- Is supplement to
- Dataset: 10.5281/zenodo.18904146 (DOI)