The Public Utility Data Liberation (PUDL) Project

See the release notes for v2022.11.30 in our docs for a more narrative explanation of what has changed.

What's Changed

This is all the PRs that were merged since the last release, excluding those made by the @dependabot and @pre-commit-ci bots.

• Apply black autoformatting by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1543
• Apply black formatting by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1548
• Update to pip 22, setuptools 61. Add nbconvert to pudl-dev by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1565
• Add installation_year and construction_year to PPL by @katie-lamb in https://github.com/catalyst-cooperative/pudl/pull/1554
• Modify EPA CEMS ETL to facilitate Intake Catalog by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1563
• Hub EIA transition by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/1575
• Add office hours scheduling links to README by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1582
• Rename tox virtualenv dir from .env_pudl to .env_tox by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1586
• Add rstcheck to our collection of linters by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1587
• Bring in year_state_filter tests & improvements from pudl_catalog by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1589
• Update maximum allowed version of setuptools to 62. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1590
• Use partial function in map of EPA CEMS ETL by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1591
• Additional code formatting/linting without modernizing Python syntax by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1598
• Refactor labeling of true granularities with plant part to generator match function by @katie-lamb in https://github.com/catalyst-cooperative/pudl/pull/1447
• Re-gigger backfilling technology_description & make prime_mover_code an annually harvested column by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/1600
• Remove some seldom used dependencies from pudl-dev environment.yml by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1615
• Fix breakage resulting from dask v2022.4.2 by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1618
• Cinco de Mayo 🇲🇽 by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1616
• Clean operational_status_code using metadata encoder by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/1624
• add opex_nonfuel column to all FERC1 plant tables in output layer for all_plants_ferc1 table by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/1626
• Add installation_year and construction_year as plant part level by @katie-lamb in https://github.com/catalyst-cooperative/pudl/pull/1578
• Add ML for sustainable energy citation to bibliography by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1641
• Dependabot auto merge by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1655
• Small docs updates by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/1642
• Dynamically generate RSTs with new DataSource metadata by @katie-lamb in https://github.com/catalyst-cooperative/pudl/pull/1532
• Add DataSource Metadata for EPA-EIA Crosswalk by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/1676
• Address issue where 861 ETL fails w/o all years of data by @arengel in https://github.com/catalyst-cooperative/pudl/pull/1671
• update ferc-eia glue with fixes found from the FERC plant-ID-er by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/1678
• Rework clean_merge_asof func by @katie-lamb in https://github.com/catalyst-cooperative/pudl/pull/1550
• Switch to the Furo Sphinx theme by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1680
• Require Python 3.10 and update to modern syntax by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1685
• Release notes for date_merge and default columns change for PPL and MCOE by @katie-lamb in https://github.com/catalyst-cooperative/pudl/pull/1690
• Apply Yaml pre-commit formatter by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/1689
• Fix all plants ferc1 by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/1656
• Require Python 3.10 in the pudl-dev conda environment by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1697
• Draft: add capacity mw to mcoe defaults and fix row counts in validation tests for eia tables by @katie-lamb in https://github.com/catalyst-cooperative/pudl/pull/1695
• GCE Deploy by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1627
• Add workflow_dispatch support to nightly builds by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/1702
• Add build-deploy-pudl.yml to main by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/1703
• Fix Github Ref bug by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/1704
• Unpin apt-get packages by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/1725
• Remove unnecessary packages from build system; specify backend. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1743
• Update setuptools numpy by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1745
• Add gcs and bypass cache args to datastore cli by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/1740
• Fix dependabot automerge by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1753
• Bring new bot-auto-merge workflow into main by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1756
• Move slowly varying plant attributes from entity to annual plants table by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1749
• Fix bug in gens mega and plant part list creation by @katie-lamb in https://github.com/catalyst-cooperative/pudl/pull/1759
• Integrate EPA CEMS hourly emissions data for 2021 by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1778
• EIA923 early release, EIA860 early release, and 860m 2022-06 by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/1834
• Avoid using Shapely v1.8.3 due to upstream bug / incompatibility by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1848
• Update eia923 raw inputs to include revisions made by EIA on 2022-08-11 by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1846
• Add missing columns and update EIA860, EIA860M and EIA923 data for 2021 by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/1836
• Use gcs cache in ci by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1858
• Patch nightly build flakiness by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/1856
• Create a data_maturity label for EIA data by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/1855
• Update bug report issue template by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1869
• Update ETL settings files to work with XBRL+DBF and new Ferc1Settings by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/1886
• Xbrl steam but really by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/1881
• Use internal zenodo-cache bucket for nightly builds by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/1880
• Encode balancing authority codes by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/1897
• Fill in some null BA codes using BA names by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/1906
• Implement drop_invalid_rows() for fuel_ferc1 table by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1903
• Split TableTransformer.transform() into 3 phases by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1900
• Prepare raw FERC XBRL DB's for publication with Datasette by @zschira in https://github.com/catalyst-cooperative/pudl/pull/1831
• Aggregate data_maturity in gfn_eia923, update EIA ETL debugging Notebook by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1915
• Use provision-micromamba and remove ferc1_solo ETL to speed up CI. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1913
• Fill in pre-2013 BA codes by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/1911
• Update the name of the EPA CAMD to EIA crosswalk data source. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1918
• Add metadata & DOIs for EIA Bulk Electricity data source by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1922
• Integrate EIA-861 2021 Early Release data by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1921
• Updating 861 package_data for 2021 early release by @arengel in https://github.com/catalyst-cooperative/pudl/pull/1920
• Add epacems crosswalk to etl by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/1692
• Re-add and update the epacamd-eia crosswalk analysis module by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/1934
• Add updated crosswalk analysis back into dev by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/1938
• Ensure PUDL works with Pandas 1.5.0 by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1902
• Plant part updates to fix RMI CI memory issues by @katie-lamb in https://github.com/catalyst-cooperative/pudl/pull/1865
• Fix build error for epacamd_eia_test by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/1940
• Create github action to sync zenodo caches by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/1935
• Rollback pandas-1.5 by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/1945
• Automate docstring formatting by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1948
• New FERC 1 / XBRL Transform Unit Tests by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1919
• Add a static table of political subdivision attributes. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1966
• Add release notes about the new political_subdivisions table. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1967
• Integrate xbrl to dbf utility ID mapping by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/1931
• Refactor FERC 1 transform; integrate XBRL for fuel & steam tables by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/1721
• Remove derelict docker build workflow file by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2002
• Add labeling functions for manual maps by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/1988
• EIA API Replacement by @TrentonBush in https://github.com/catalyst-cooperative/pudl/pull/1998
• EIA Bulk Electricity Aggregates by @TrentonBush in https://github.com/catalyst-cooperative/pudl/pull/1937
• Add 2021 861 to output by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/1990
• Integrating 2021 FERC1 XBRL hydro table by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/1992
• Document new transforms & organize imports/logging by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1962
• Add release notes about EIA API removal / fuel price filling by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2008
• Remove EXPERIMENTAL from datasette title by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/2010
• Boiler Fuel Allocation Improvements by @grgmiller in https://github.com/catalyst-cooperative/pudl/pull/1608
• Bump allowed geopandas & pygeos conda versions by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2019
• Add 2021 eia923 eia860 final release by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/2017
• Docs housekeeping by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2039
• Add a draft PR template. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2038
• Fix erroneous max row values in mcoe_test by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/2041
• Allocate fuel consumption by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/1096
• Integrate pudl ID's from eia final updates into the xbrl branch by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/2042
• Updated docs to include nightly build failure etiquette. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2044
• remove pesky rounds on floats by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/2052
• xbrl and dbf integration of pumped storage ferc1 table by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/2005
• xbrl and dbf integration of purchased power ferc1 table by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/2011
• First draft of plant_in_service_ferc1 DBF + XBRL transform by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2025
• Stop warning about duplicate record IDs & missing columns that don't matter by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2054
• track down and fix warnings in xbrl transform by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/2057
• Fix bad multi-index construction that was scrambling XBRL columns. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2059
• Transforms for small gens table XBRL by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/2035
• Integrate XBRL taxonomy metadata into plant_in_service transform by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2058
• add release notes for xbrl integration by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/2063
• Xbrl nov archive by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/2065
• Fix doc stings for small gens table transform by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/2068
• Resolve duplicate DBF-XBRL mappings and duplicate records in plant_in_service_ferc1 by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2067
• accumulation of any changes in the xbrl validation tests by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/2071
• Initial FERC Form 1 XBRL Integration by @zschira in https://github.com/catalyst-cooperative/pudl/pull/1665
• Address missing fuel ferc1 records by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2092
• Add loglevel and logfile to all CLI commands by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/2091
• Add FERC XBRL DBs to datasette deployment. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2095
• Release 2022.11.XX by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1681
• Add GCP region/zone env var to nightly build scripts by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/2101
• Packaging & release note tweaks for PUDL v2022.11.30 by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2098
• fix missing generator_id by @grgmiller in https://github.com/catalyst-cooperative/pudl/pull/2122
• v2022.11.30 by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2135

New Contributors

• @arengel made their first contribution in https://github.com/catalyst-cooperative/pudl/pull/1671
• @grgmiller made their first contribution in https://github.com/catalyst-cooperative/pudl/pull/1608

Full Changelog: https://github.com/catalyst-cooperative/pudl/compare/v0.6.0...v2022.11.30