Historical Observations Data Platform Weather Stations

The Historical Data Platform (HDP) is a cloud-based, historical weather observations data platform focusing on near-real time to enable California’s energy sector access to high-quality, open climate and weather data in the formats that energy practitioners require. HDP applies best-in-class standards to historical weather observations to increase the number of reliable, accurate, long-standing observational records available to energy sector planners, including 8 variables identified as critical by energy industry engagement. HDP’s QA/QC pipeline included variable expansion (e.g., solar radiation, winds, precipitation) and careful handling for the kinds of extreme events facing western North America. The QC protocol is comprehensive, scientific, open, and transparent. Care was taken to address extreme events and ensure that the quality control process did not remove real extreme events while fixing issues such as sensor malfunctions. Inaccurate observations can cause serious problems for energy practitioners, leading to consequences such as incorrect maintenance decisions, missed safety thresholds, or over- or under-investment in grid hardening. This dataset supports ongoing efforts in electricity demand planning, asset placement, resiliency, and estimating renewable energy generation capacity needs, and serves as a critical localized dataset for future downscaling efforts of global climate models in the variables and resolution (hourly, sub 3km) needed in energy sector applications. The HDP dataset includes 15,064 standardized stations sourced from 27 open-access networks and covers 1980-2022, increasing the number of high-quality weather observing stations available to data users from ~500 to 15,000.

 

Dataset Format Note: Each station is stored as an independent .zarr in AWS S3 cloud storage. Open-access pathway to individual stations is provided in the historical_wx_station.csv catalog. Please see the weather_station_data_access.ipynb Jupyter Notebook (Python) for access. 

 

This dataset is included in California’s Fifth Climate Change Assessment.