PUDL v2024.02.05 Data Release

Data Release Notes

This release contains only minor data updates compared to what we put out in December, however the database naming conventions and release process has changed pretty dramatically. We are confident these changes will make the data we publish more accessible, and allow us to push out updates much more frequently going forward.

We also finally merged in improvements and generalizations to our record linkage processes, which were generously supported by a CCAI Innovation Grant. Connecting disparate public datasets that describe the same physical infrastructure and corporate entities is one of the most valuable improvements we make to the data, and we are excited to be able to be able to do it in a more general, reproducible way so we can easily apply it to other datasets. We’ve already started work on a Mozilla Foundation grant to link SEC data to the FERC and EIA data we already have, allowing us to track ownership relationships between utility holding companies and their many subsidiaries. We expect the same kind of process will be useful for linking the PHMSA gas pipeline data to natural gas utilities that report to EIA and FERC.

Database Naming Conventions

Our main focus with this release was to overhaul the naming system for our nearly 200 database tables. This will hopefully make it easier to find what you’re looking for, especially if you are a new PUDL user. We think it will also make it easier for us to keep the database organized as we continue to expand its scope. For an explanation of the new naming conventions, see Naming Conventions, and to see the full list of all available tables, see the PUDL Data Dictionary.

This is a major breaking change for anybody is accessing the database directly. Stick with the v2023.12.01 release until you’re ready to update your references to the old database table names. For the time being we have patched the old pudl.output.pudltabl.PudlTabl class so that it behaves as similarly as possible to before. However, we plan to remove this output class in the near future, and no new database tables will be made accessible through it. Going forward we expect users to use the database directly, freeing them from the need to install all of the software and dependencies which we use to produce it, hopefully improving the data’s technical accessibility and platform independence.

For more development details see #2765 which was the main epic tracking this process (with many sub-issues: #2777, #2788, #2812, #2868, #2992, #3030, #3173, #3174, #3223) and PR #2818.

Record Linkage Improvements

• The pudl.analysis.record_linkage.eia_ferc1_record_linkage module has been refactored substantially to make use of more generic PUDL record linkage infrastructure and include extra cleaning steps. This resulted in around 500 or 2% of matches changing. See catalyst-cooperative/ccai-entity-matching#108 and #3184.

• Update the FERC Form 1 plant ID assignment (Identifying related plant records from different years within the FERC Form 1 data) to use the new record linkage infrastructure. See #3007, #3137

New Data Coverage

• Updated EPA Hourly Continuous Emission Monitoring System (CEMS) to switch to pulling the quarterly updates of CEMS instead of the annual files. Integrates CEMS through 2023Q3. See issue #2973 & PR #3096, #3139.

• Began integration of PHMSA gas distribution and transmission tables into PUDL, extracting raw data from 1990-present. Note that these tables are not yet being written to the database as they are still raw. See epic #2848, and constituent PRs: #2932, #3242, #3254, #3260, #3262, #3266, #3267, #3269, #3270, #3279, #3280.

• We began integration of data from EIA Forms 176, 191, and 757, describing natural gas sources, storage, transporation, and disposition. Note this data is still in its raw extracted form and is not yet being written to the PUDL DB. See #3304, #3227

• Updated the EIA Bulk Electricity data archive so that the available data now to runs through 2023-10-01. See #3252. Also added this dataset to the set of data that will automatically generate archives each month. See This PUDL Archiver PR and this Zenodo archive

Data Cleaning

• Filled in null annual balances with fourth-quarter quarterly balances in core_ferc1__yearly_balance_sheet_liabilities_sched110. #3233 and #3234.

• Added a notebook devtools/debug-column-mapping.ipynb to make debugging manual column maps for new datasets simpler and faster.

Metadata Cleaning

• Fix metadata structures and pyarrow schema generation process so that all tables can now be output as Parquet files. See issue #3102 and PR #3222.

• Made a description field mandatory for all instances of Field and Resource. Updated the pudl.metadata.fields.FIELD_METADATA` and pudl.metadata.resources.RESOURCE_METADATA` so that all of them have a description. This primarily affected EIA Form 861 – Annual Electric Power Industry Report tables. See #3224, #3283.

• Removed fields that are not used in any tables and removed the xfail from the test_defined_fields_are_used test. #3224, #3283.

Other PUDL v2024.02.05 Resources

• PUDL v2024.02.05 Data Dictionary
• PUDL v2024.02.05 Documentation
• PUDL in the AWS Open Data Registry
• PUDL v2024.02.05 in a free, public AWS S3 bucket: s3://pudl.catalyst.coop/v2024.02.05/
• PUDL v2024.02.05 in a requester-pays GCS bucket: gs://pudl.catalyst.coop/v2024.02.05/
• Zenodo archive of the PUDL GitHub repo for this release
• PUDL v2024.02.05 release on GitHub
• PUDL v2024.2.5 package in the Python Package Index (PyPI)

Contact Us

If you're using PUDL, we would love to hear from you! Even if it's just a note to let us know that you exist, and how you're using the software or data. Here's a bunch of different ways to get in touch:

• Follow us on GitHub
• Use the PUDL Github issue tracker to let us know about any bugs or data issues you encounter
• GitHub Discussions is where we provide user support.
• Watch our GitHub Project to see what we're working on.
• Email us at hello@catalyst.coop for private communications.
• On Mastodon: @CatalystCoop@mastodon.energy
• On BlueSky: @catalyst.coop
• On Twitter: @CatalystCoop
• Play with our data and notebooks on Kaggle
• Combine our data with ML models on HuggingFace
• Learn more about us on our website: https://catalyst.coop
• Subscribe to our announcements list for email updates.