Group Terrestrial Ecohydrology @ Uni Jena and UFZ Leipzig
Published September 3, 2025 | Version v1
Computational notebook Open

Quality control and flagging of soil moisture time series data at the Global Change Experimental Facility (GCEF), Bad Lauchstädt

Creators

  • 1. ROR icon Helmholtz Centre for Environmental Research
  • 2. ROR icon Friedrich Schiller University Jena
  • 3. Helmholtz-Zentrum fur Umweltforschung Department Monitoring- und Erkundungstechnologien
  • 4. ROR icon German Centre for Integrative Biodiversity Research

Description

This repository contains the Python scripts for the workflow for quality control, flagging, and correcting soil moisture time series data gathered on grasslands at the Global Change Experimental Facility (GCEF) in Bad Lauchstädt, Central Germany. The workflow is explained in a manuscript submitted and in review to Scientific Data. It consists of the following steps:

  1. Basic quality control (Dorigo et al., 2021, Schmidt et al., 2023)
    (a) Flag missing data
    (b) Flag values outside the range 0-100 vol% as "bad"
    (c) Flag values outside the range 2-60 vol% as "doubtful" (-60 to +60 °C for soil temperature)
    (d) Search for positive and negative jumps and flag their index positions as "doubtful"
    (e) Flag constant values as "bad": volumetric soil water content, if exceeding 7 d; soil temperature, if exceeding 6 h
  2. Cross flagging: Flag values where soil temperature is below zero as "doubtful" (Dorigo et al., 2021)
  3. Flag all non-flagged values as "OK"
  4. Apply jump correction and flag accordingly
  5. Apply correction for temperature signal and flag accordingly
  6. Smoothing of the time series and flag accordingly

The temperature correction (step 5) is loosely based on Kapilaratne & Lu (2017). The newly realized procedure for jump correction (step 4), is explicitly documented in Jupyter notebooks (folder "jupyter").

The code published here was used in and is described in detail in Westermann et al. (sub.). Datasets processed with and resulting from these scripts can be found in https://doi.org/10.5281/zenodo.17054266.

The following folders are provided:

  • jupyter: contains Jupyter notebooks of the established correction procedures for jumps in the time series and for temperature correlation of the soil moisture values, shown with example time series
  • gcef-pipeline: contains configuration files for setting up and executing the snakemake pipeline for quality control of the soil moisture time series data, which uses the scripts in the folder scripts. More details of usage, including required code adjustments to run the code on a different platform, can be found in the Readme.md
  • sm_processing: contains the script for merging the data of all sensors from one plot and adding flags of raw and corrected data

Files

GCEF_SoilMoisture_QC.zip

Files (5.1 MB)

Name Size Download all
md5:cdb34d8efaa38f1b26b0e2b5c5c0f813
5.1 MB Preview Download

Additional details

Software

Programming language
Python
Development Status
Active

References

  • Dorigo, W., Himmelbauer, I., Aberer, D., Schremmer, L., Petrakovic, I., Zappa, L., Preimesberger, W., Xaver, A., Annor, F., Ardö, J., Baldocchi, D., Bitelli, M., Blöschl, G., Bogena, H., Brocca, L., Calvet, J.-C., Camarero, J. J., Capello, G., Choi, M., Cosh, M. C., van de Giesen, N., Hajdu, I., Ikonen, J., Jensen, K. H., Kanniah, K. D., de Kat, I., Kirchengast, G., Kumar Rai, P., Kyrouac, J., Larson, K., Liu, S., Loew, A., Moghaddam, M., Martínez Fernández, J., Mattar Bader, C., Morbidelli, R., Musial, J. P., Osenga, E., Palecki, M. A., Pellarin, T., Petropoulos, G. P., Pfeil, I., Powers, J., Robock, A., Rüdiger, C., Rummel, U., Strobel, M., Su, Z., Sullivan, R., Tagesson, T., Varlagin, A., Vreugdenhil, M., Walker, J., Wen, J., Wenger, F., Wigneron, J. P., Woods, M., Yang, K., Zeng, Y., Zhang, X., Zreda, M., Dietrich, S., Gruber, A., van Oevelen, P., Wagner, W., Scipal, K., Drusch, M., and Sabia, R.: The International Soil Moisture Network: serving Earth system science for over a decade, Hydrology and Earth System Sciences, 25, 5749–5804, https://doi.org/10.5194/hess-25-5749-2021, 2021.
  • Kapilaratne, R. G. C. J. and Lu, M.: Automated general temperature correction method for dielectric soil moisture sensors, Journal of Hydrology, 551, 203–216, https://doi.org/10.1016/j.jhydrol.2017.05.050, 2017.
  • Schmidt, L., Schäfer, D., Geller, J., Lünenschloss, P., Palm, B., Rinke, K., Rebmann, C., Rode, M., and Bumberger, J.: System for automated Quality Control (SaQC) to enable traceable and reproducible data streams in environmental science, Environmental Modelling Software, 169, 105 809, https://doi.org/10.1016/j.envsoft.2023.105809, 2023.