International Soil Moisture Network (ISMN) Data Processing Pipeline

ISMN Data Extraction and Formatting Pipeline

This repository provides an efficient pipeline to batch process, filter, and standardize soil moisture observation data from the International Soil Moisture Network (ISMN) into daily analysis-ready CSV files, including detailed site metadata.


🚀 Key Features

✅ Automated Directory Scanning and Indexing**:

  * Recursive scanning of the ISMN root directory (organized by network → station → file).
  * Automatic caching of site metadata to accelerate future processing (`./tmp/info.csv`).

✅ Advanced Multi-dimensional Filtering**:

  * Filter sites by continent, network (e.g., USCRN).
  * Depth-based filtering supporting three thresholds (`0.05`, `0.1`, `0.2` m ...).

✅ Robust Data Quality Control**:

  * Observations outside the range `[0,1]` are flagged as missing (`NaN`).
  * Aggregates hourly data to daily averages.

✅ Flexible Data Export**:

  * Annual CSV outputs (`soil_moisture.csv`) organized by year (rows: sites, columns: daily values).
  * Detailed site coordinates (`crd.csv`) and metadata (`info.csv`) included.



If you use this pipeline in your research, please cite:

> Liu, J., Rahmani, F., Lawson, K., & Shen, C. (2022). A multiscale deep learning model for soil moisture integrating satellite and in situ data. Geophysical Research Letters, 49(7), e2021GL096847. [https://doi.org/10.1029/2021GL096847](https://doi.org/10.1029/2021GL096847)

> Liu, J., Hughes, D., Rahmani, F., Lawson, K., & Shen, C. (2023). Evaluating a global soil moisture dataset from a multitask model (GSM3 v1.0) with potential applications for crop threats. Geoscientific Model Development, 16(5), 1553–1567. [https://doi.org/10.5194/gmd-16-1553-2023](https://doi.org/10.5194/gmd-16-1553-2023)

---