Density assimilated reduced order state for thermosphere-ionosphere drag estimation

Data product accompanying the paper Reduced-Order Data Assimilation for Thermospheric Density Using Physics-informed SINDyc Models.

Assimilated reduced-order thermospheric density state, 10 PCA modes on a 60 s time base, spanning 2000-08-01 00:00:00 to 2025-11-30 23:59:00 UTC (13,324,320 steps across 309 assimilation windows).

The product file is deposited in six parts

pca_ark_results.h5 is 4.92 GB and is split into six sequential parts, pca_ark_results.h5.part-000 through pca_ark_results.h5.part-005. Download all six into one directory and join them in order:

./reassemble.sh                              # joins, then verifies the checksum
cat pca_ark_results.h5.part-* > pca_ark_results.h5    # equivalent

The split is on raw bytes, so the joined file is identical to the original — there is no unpacking step, and the individual parts are not separately readable. The joined file must have SHA-256 45da93ad492d2f97d1ee4824d8cc59bd0113811a2b6a75a46daf1b749878a6e3. SHA256SUMS.txt also lists a checksum for every individual part, so a truncated download can be identified without re-joining. Verify before use: an incomplete part produces a file that opens but returns malformed data.

Contents of pca_ark_results.h5

• timestamps — UTC timestamp of each filter step, as fixed-format YYYY-MM-DD HH:MM:SS byte strings. (13,324,320,)
• z_filt — posterior (filtered) ROM state coefficients. (10, 13,324,320) float64
• z_openloop — free-running ROM state coefficients, no measurement updates, for baseline comparison. (10, 13,324,320) float64
• window_log — per-window assimilation summary. (309,)
• steplog/ — per-step Kalman filter diagnostics:
  • trace_P — trace of the posterior covariance (the A-optimal scalar).
  • diag_P — diagonal of the posterior covariance (marginal variances of each ROM coefficient). (13,324,320, 10)
  • NIS — Normalised Innovation Squared, for filter consistency checks.
  • innov_rms — RMS of the innovation vector.
  • n_meas — number of measurements assimilated at each step (0–4).
  • sat_log — satellites contributing measurements, one entry per assimilating step.
  • mode — filter operating mode: 1 = measurement update applied, 0 = propagation only.

Indexing

All step-indexed arrays are aligned with timestamps except sat_log, which has 13,176,643 entries — one per step with mode == 1, omitting the 147,677 propagation-only steps. Each entry is self-describing, prefixed with its within-window step index, e.g. 18715:grace-fo=1,swarm_a=1,swarm_b=1,swarm_c=1. Index it via that prefix or via mode == 1, not positionally against timestamps.

NIS is NaN exactly where n_meas == 0 (1.11% of steps), which is the same set as mode == 0. Mask these before computing consistency statistics.

Other files

• best_storms_uu_th_0.01_ridge_122.0_aligned.mat — the PCA basis (U, mu) the reduced state is expressed in. Required to reconstruct a full density field as rho = mu + U @ z; not required to read z_filt / z_openloop on their own.
• pca_ark_results_metadata.json — variable list, grid definition, derived process-noise vector, P0, model file, and the measurement-noise convention.
• density_reconstruction.ipynb — reconstructs gridded density at any time and location in the dataset span. Self-contained: the spatial interpolator is inlined, so the only requirements beyond the files above are h5py, numpy, scipy and matplotlib. Ships pre-executed on the Halloween 2003 storm window as a worked example.
• reassemble.sh, SHA256SUMS.txt — reassembly and integrity verification.
• density_reconstruction_summary.png — summary figure.

For the assimilation code itself, see https://doi.org/10.5281/zenodo.19947885.