June 30, 2019 Dataset Open Access

Gradient Boosting Machine Learning to Improve Satellite-Derived Column Water Vapor Measurement Error

Allan C. Just; Yang Liu; Meytar Sorek-Hamer; Johnathan Rush; Michael Dorman; Robert Chatfield; Yujie Wang; Alexei Lyapustin; Itai Kloog

The atmospheric products of the Multi-Angle Implementation of Atmospheric Correction (MAIAC) algorithm include column water vapor (CWV) at 1 km resolution, derived from daily overpasses of NASA’s Moderate Resolution Imaging Spectroradiometer (MODIS) instruments aboard the Aqua and Terra satellites. We have recently shown that machine learning using extreme gradient boosting (XGBoost) can improve the estimation of MAIAC aerosol optical depth (AOD). Although MAIAC CWV is generally well validated (Pearson’s R >0.97 versus CWV from AERONET sun photometers), it has not yet been assessed whether machine-learning approaches can further improve CWV. Using a novel spatiotemporal cross-validation approach to avoid overfitting, our XGBoost model with nine features derived from land use terms, date, and ancillary variables from the MAIAC retrieval, quantifies and can correct a substantial portion of measurement error relative to collocated measures at AERONET sites (27.8% and 15.5% decrease in Root Mean Square Error (RMSE) for Terra and Aqua datasets, respectively) in the Northeastern USA, 2000-2015. We use machine-learning interpretation tools to illustrate complex patterns of measurement error and describe a positive bias in MAIAC Terra CWV worsening in recent summertime conditions. We validate our predictive model on MAIAC CWV estimates at independent stations from the SuomiNet GPS network where our corrections decrease the RMSE by 17% and 8% for Terra and Aqua MAIAC CWV. Empirically correcting for measurement error with machine-learning algorithms is a post-processing opportunity to improve satellite-derived CWV data for Earth science and remote sensing applications.

# About the attachment #

1. The first zip file contains the data (collocated datasets) alone. The datasets in the zip file are JSON files that can be opened directly in browsers, text editor, or R using functions like `jsonlite::fromJSON`.

2. The second zip file (CWV-project-repository) is an R project contains all the code (/Code) and data (/Data) needed to reproduce results. 
The folder (/Intermediate) contains the intermediate cross-validation modeling results. If initiating R project using the _cwv_paper.Rproj_, the Rmarkdown file (mainly in _03_cwv_10by10cv_resultsmd.Rmd_) can reproduce all the results (figures and tables) used in the paper.