Degradation of Automatic Dependent Surveillance-Broadcast Integrity: Temporal Clustering and Anomaly Persistence
Description
Degradation of Automatic Dependent Surveillance-Broadcast Integrity: Temporal Clustering and Anomaly Persistence
Data License: Creative Commons Attribution 4.0 International (CC BY 4.0) Code License: MIT
Overview
Welcome to the replication dataset for the study "Degradation of Automatic Dependent Surveillance-Broadcast Integrity: Temporal Clustering and Anomaly Persistence". This repository contains the complete open-science codebase, the daily extraction shards, the pre-computed survival matrices, and the statistical artifacts required to fully reproduce the longitudinal analysis of 2025 global ADS-B tracking integrity.
Extracting from a gross volume of 246.7 billion OpenSky Network ADS-B messages logged across the 2025 calendar year, this study analyzes 229.0 billion filtered, airborne state vectors. The out-of-core computational pipeline executes scrambled-temporal permutation testing ($\mathcal{W}={3,5,7,10,15}$), day-resolution time-to-first-anomaly Kaplan-Meier survival modeling under two explicit censoring conventions, exposure-adjusted Cox proportional hazards regression, and an Andersen-Gill recurrent-event model, quantifying longitudinal surveillance integrity across 233,859 unique aircraft trajectories.
Version 2 rebuilds the survival analysis at day resolution and corrects five implementation defects found in the Version 1 survival scripts during peer-review revision. The clustering pipeline is unchanged, and a from-scratch re-execution reproduced its aggregates bit-identically. See CHANGELOG.md for the full list. Superseded Version 1 scripts and outputs remain available under Version 1 of this record.
Quick Start & Reproducibility
To replicate the pipeline locally:
- Install dependencies:
pip install -r requirements.txt(Python 3.10+; the pipeline auto-throttles worker parallelism and runs on machines with 8 GB of RAM). - Execute the pipeline. All stages below run entirely from the daily shards in
DATA/extractions/; the shards are the Stage 1 extraction outputs, so no OpenSky credentials are required.CODE/stage2_data_generation.py: The Version 1 clustering engine (Numba JIT permutation tests). Optional: its results are pre-computed inDATA/H2_enhanced_results.parquet.CODE/stage2b_daygrain_survival_generation.py: Builds the day-grain survival cohort from the daily ledgers (gap-anchored primary convention; event-priority sensitivity).CODE/stage2c_exposure_recurrence_models.py: Cox time-to-first model, Andersen-Gill recurrent-event model, placebo suite, and ridge-penalizer sweep.CODE/stage2d_null_model_diagnostics.py: Injected-independence null calibration and index-space diagnostics.CODE/stage2b_s2_threshold_sweep.py: Dropout-censoring threshold sweep (15/30/45/60 days) with an internal identity gate against the main cohort.CODE/stage3b_daygrain_metrics_export.py: Exports every manuscript metric and figure-ready curve.CODE/stage3_figure_generation.py: Re-renders the manuscript figure PDFs with an internal integrity gate.
Independent re-execution of this package reproduced the manuscript's block-sensitivity aggregates and the permutation-validation agreement bit-identically.
Directory Structure
.
├── README.md
├── CHANGELOG.md
├── requirements.txt
├── CODE/
│ ├── config.py
│ ├── extract_manuscript_data_v2.py
│ ├── stage2_data_generation.py
│ ├── stage2b_daygrain_survival_generation.py
│ ├── stage2b_s2_threshold_sweep.py
│ ├── stage2c_exposure_recurrence_models.py
│ ├── stage2d_null_model_diagnostics.py
│ ├── stage3b_daygrain_metrics_export.py
│ └── stage3_figure_generation.py
├── DATA/
│ ├── H2_enhanced_results.parquet
│ └── extractions/
│ ├── Dataset_2A_aggregated_20250101..20251231.parquet (365 daily shards)
│ ├── Dataset_2B_ledger_20250101..20251231.parquet (365 daily shards)
│ └── Q1_diagnostic_stats_20250101..20251231.csv (365 daily reports)
├── LOGS/
│ ├── stage2_data_generation.log
│ ├── stage2b_daygrain_survival.log
│ ├── stage2b_s2_threshold_sweep.log
│ ├── stage2c_exposure_recurrence.log
│ ├── stage2d_null_diagnostics.log
│ ├── stage3b_daygrain_metrics.log
│ ├── stage3_figure_generation.log
│ └── stage3_figure_generation_v13.log
└── RESULTS/
├── Master_Survival_Cohort_DayGrain.parquet
├── Master_Survival_Cohort_DayGrain_d15.parquet
├── Master_Survival_Cohort_DayGrain_d45.parquet
├── Master_Survival_Cohort_DayGrain_d60.parquet
├── ledger_daygrain_monthly.parquet
├── df_ledger.parquet
├── aux_max_internal_gap_days.parquet
├── model_cox_daygrain_coefficients.csv
├── model_cox_daygrain_month1_excluded.csv
├── model_ag_recurrent_coefficients.csv
├── model_penalizer_sweep_daygrain.csv
├── model_placebo_null_distribution.csv
├── model_placebo_summary.json
├── model_schoenfeld_daygrain.txt
├── Table_S3_block_preserving_sensitivity_multi_w.csv
├── tab_stratified_clustering_rates.csv
├── tab_censoring_demographics_daygrain.csv
├── tab_logrank_daygrain.csv
├── tab_buffer_share.csv
├── tab_dt_translation.csv
├── tab_window_concordance.csv
├── tab_negative_control.csv
├── tab_replacement_sensitivity.csv
├── data_permutation_validation.csv
├── daygrain_persistence_summary.json
├── Additional_Manuscript_Metrics_DayGrain.json
├── manuscript_fill_values_v2.0.json
├── fig_daygrain_survival_data.csv
├── fig_daygrain_severity_strata_data.csv
├── Figure_2_daygrain_severity_onset_survival.pdf
├── Figure_3_daygrain_exposure_tertiles.pdf
├── Figure_S1_censoring_design_sensitivity.pdf
├── Figure_S2_dropout_threshold_sensitivity.pdf
├── Figure_S3_penalizer_stability.pdf
├── Figure_S4_continental_subset.pdf
├── Figure_S5_null_calibration.pdf
└── Figure_S6_block_retention_stratified.pdf
File Manifest & Data Dictionary
1. Root Files
README.md: This documentation file.CHANGELOG.md: What Version 2 adds and corrects, and what remains unchanged.requirements.txt: Pinned Python dependencies for environment reproducibility.
2. CODE/ (The Computational Pipeline)
config.py: Configuration ledger V1.1. All kinematic thresholds (e.g., speed $> 6670.3$ m/s), censoring parameters (30-day dropout threshold, both censoring conventions), penalizer grid, memory-guard limits, window scales, and cryptographic salts in one audited module.extract_manuscript_data_v2.py: Parses the pipeline outputs to extract and format every macro-statistic referenced in the manuscript text.stage2_data_generation.py: The Version 1 clustering engine, retained unchanged. Runs the out-of-core scrambled-temporal permutation tests via Numba JIT compilation; its pre-computed results ship inDATA/H2_enhanced_results.parquet.stage2b_daygrain_survival_generation.py: Reconstructs each aircraft's daily observation calendar from the ledgers and builds the day-grain survival cohort under both censoring conventions.stage2b_s2_threshold_sweep.py: Re-derives the cohort at 15/30/45/60-day dropout thresholds; its internal gate verifies the 30-day pass is row-identical to the main cohort.stage2c_exposure_recurrence_models.py: Fits the exposure-adjusted Cox model, the Andersen-Gill recurrent-event model over the aircraft-month panel, the covariate-permutation placebo suite (100 refits), and the ridge-penalizer sweep.stage2d_null_model_diagnostics.py: The injected-independence calibration study quantifying the deployed test's unconditional false-positive rate, plus buffer-share, timestamp-window, and sampling diagnostics.stage3b_daygrain_metrics_export.py: Exports all survival, persistence, and model metrics plus figure-ready curve coordinates.stage3_figure_generation.py: Renders the publication PDFs; refuses to run if its inputs fail the built-in integrity gate.
3. DATA/ (Inputs; unchanged from Version 1)
H2_enhanced_results.parquet: Pre-computed permutation-test results (Z-statistics, O/E ratios, FDR-corrected p-values for all $\mathcal{W}$).extractions/: The 365 daily shards behind the 229.0 billion evaluated state vectors: aggregated anomaly vectors (Dataset_2A_aggregated_*), the severity-classified anomaly ledger (Dataset_2B_ledger_*), and daily data-quality reports (Q1_diagnostic_stats_*). These shards are the direct inputs to the Version 2 day-grain pipeline.
4. LOGS/
Timestamped execution logs providing an audit trail of hardware performance, memory-guard triggers, cohort sizes, and model diagnostics: the Version 1 clustering and figure logs plus the Version 2 day-grain generation, threshold-sweep, modeling, diagnostic, metrics, and figure logs.
5. RESULTS/ (Final Datasets, Tables, Models, and Figures — Version 2)
Primary Datasets (.parquet):
Master_Survival_Cohort_DayGrain.parquet: The core day-resolution survival dataset (233,859 aircraft). Carries both censoring conventions: gap-anchored (primary; 131,067 events, 56.0%) and event-priority (sensitivity; 164,770 events), with pre-event exposure and observability-coverage covariates.Master_Survival_Cohort_DayGrain_d15/d45/d60.parquet: Threshold-sweep cohorts.ledger_daygrain_monthly.parquet: The aircraft-month panel behind the Andersen-Gill recurrent-event model (1,146,596 aircraft-months).df_ledger.parquet: Reconciled anomaly ledger mapping raw anomalies to their physical Severity Classes.aux_max_internal_gap_days.parquet: Per-aircraft maximum internal observation gap (drives the continental subset).
Statistical Models & Validation:
model_cox_daygrain_coefficients.csv: Cox time-to-first outputs at $\lambda = 0.1$.model_cox_daygrain_month1_excluded.csv: The first-interval-excluded sensitivity fit.model_ag_recurrent_coefficients.csv: Andersen-Gill recurrent-event outputs (655,588 recurrent cluster-months).model_penalizer_sweep_daygrain.csv: Coefficients across $\lambda \in {0.01, 0.05, 0.1, 0.5, 1.0}$.model_placebo_null_distribution.csvandmodel_placebo_summary.json: The covariate-permutation placebo suite (100 refits; null coefficient 0.0001, SD 0.0027).model_schoenfeld_daygrain.txt: Scaled Schoenfeld residual tests for the proportional-hazards assumption.data_permutation_validation.csv: N=200 vs. N=10,000 validation (100.0% agreement).
Diagnostics & Sensitivity Tables (.csv):
Table_S3_block_preserving_sensitivity_multi_w.csv: Per-aircraft block-preserving significance classifications across all window scales.tab_stratified_clustering_rates.csv: Utilization-stratified significance and block-retention rates.tab_censoring_demographics_daygrain.csv: Demographic profiles by survival state under both censoring conventions.tab_logrank_daygrain.csv: Log-rank tests between severity-onset cohorts.tab_buffer_share.csv: Day-boundary buffer share of the index space (mean 0.4% of slots).tab_dt_translation.csv: Inter-message interval percentiles (index-to-seconds translation).tab_window_concordance.csv: Duration-standardized window replication (Jaccard 0.99999 across 18,243,276 aircraft-days).tab_negative_control.csv: Injected-independence false-positive rates by density stratum (0.040 vs. nominal 0.05).tab_replacement_sensitivity.csv: Exact without-replacement re-test of the 500 densest aircraft (zero significance changes).
Manuscript Scalars & Curves:
Additional_Manuscript_Metrics_DayGrain.json,daygrain_persistence_summary.json,manuscript_fill_values_v2.0.json: Every dynamic scalar referenced in the manuscript body (survival probabilities, persistence endpoints, fleet totals).fig_daygrain_survival_data.csv,fig_daygrain_severity_strata_data.csv: The $X,Y$ coordinates plotted in the survival figures.
Publication Graphics (.pdf):
Figure_2_daygrain_severity_onset_survival.pdf: Time-to-first clustered anomaly by severity at onset (all 131,067 events classified).Figure_3_daygrain_exposure_tertiles.pdf: Exposure-stratified time-to-first survival.Figure_S1_censoring_design_sensitivity.pdf: Gap-anchored vs. event-priority censoring conventions.Figure_S2_dropout_threshold_sensitivity.pdf: Stability across 15/30/45/60-day dropout thresholds.Figure_S3_penalizer_stability.pdf: Cox coefficient behavior across the ridge-penalizer grid.Figure_S4_continental_subset.pdf: Continuously observed (continental) subset control.Figure_S5_null_calibration.pdf: Null calibration under injected independent anomalies.Figure_S6_block_retention_stratified.pdf: Block-preserving retention among uniform-significant aircraft, by utilization stratum.
How to Cite
If you use this dataset, pipeline architecture, or empirical baseline in your research, please cite the accompanying manuscript:
Pik, E. (2026). Degradation of Automatic Dependent Surveillance-Broadcast Integrity: Temporal Clustering and Anomaly Persistence. Reliability Engineering & System Safety (Under Review, JRESS-D-26-02922).
Dataset DOI (all versions): 10.5281/zenodo.19698274
This version (v2): 10.5281/zenodo.21878953
Notes (English)
Files
CHANGELOG_v2.md
Files
(2.6 GB)
| Name | Size | |
|---|---|---|
|
md5:939e422ad8fd7d442d435288e2466734
|
4.8 kB | Preview Download |
|
md5:21ed90b12bbc9f6aad6e0d450359eaf7
|
52.5 kB | Preview Download |
|
md5:b628844b3a35e855392fa3eca60b32fc
|
2.5 GB | Preview Download |
|
md5:17d8b8a49d0cb015a1ad6539d497e07e
|
76.3 kB | Preview Download |
|
md5:2490acfb6df97ddbcbc790cae0f226f0
|
14.0 kB | Preview Download |
|
md5:3f9ce1a650f67c6772ffb8ede1983ad7
|
360 Bytes | Preview Download |
|
md5:c6cf443de88dcf62aae697979cd7af79
|
110.5 MB | Preview Download |
Additional details
Related works
- Is supplement to
- Dataset: 10.5281/zenodo.18904146 (DOI)