Open Grid Emissions Initiative

This release primarily addresses an issue identified in https://github.com/singularity-energy/open-grid-emissions/issues/271, in which our data pipeline was dropping a substantial amount of data due to mismatches in the reported energy source codes used in EIA-860 and EIA-923, and our failure to validate the outputs of the allocation process more carefully. While fixing this issue, we also came across several other issues that were causing anomalous emission factor outputs.

Summary of changes

• Previously, some generation and fuel data reported in EIA Form 923 was being dropped from OGE due to inconsistent energy source codes being used for certain plants between the EIA-923 input data and EIA-860 input data. This was resulting in incorrect emissions and generation totals for certain plants, as well as incorrect primary fuel categories being assigned to these plants.
• This release also includes several updates to the method for identifying the primary fuel type of each plant, which fixes a bug that was causing certain nuclear plants to be identified as a non-nuclear fuel type due to missing fuel consumption data in EIA-923.
• This release also includes updates to our methodology for converting gross generation data reported in CEMS to net generation. These updates include more stringent standards for which conversion factors are used for each plant, and more robust backstop conversion factors. This update will result in more net generation being reported for certain plants, and more realistic plant-level emission intensity values.
• We have also added the newly-released eGRID2021 dataset to the list of downloaded files so that 2021 OGE values can be easily compared to 2021 eGRID values using our validation notebooks included in the repository.

Detailed changes Fixes the generation and fuel allocation process

• Fix bugs in pudl allocate_net_gen module, as described in this PR (https://github.com/catalyst-cooperative/pudl/pull/2235):

  • Adds a new function add_missing_energy_source_codes_to_gens() that adds energy_source_codes that appear in the gf table but not gens to gens.
  • In some cases, non-zero fuel consumption and net generation is reported in the EIA-923 generation and fuel table that is associated with an energy_source_code that is not associated with that plant-prime mover in the gens table, which would cause these data to get dropped when these two tables are merged. To fix this, for each plant-pm, this function identifies such esc, and adds them to the gens_at_freq table as new energy_source_code columns.
  • The sub-function identify_missing_gf_escs_in_gens() identifies when there are fuels reported in the gf table for that plant-pm that are not listed in the gens table for that plant pm.
  • Adds the MISSING_SENTINEL value to the net_generation_mwh_g_tbl column. For some reason, this column had been commented out, which was leading to NaNs appearing in the data when dividing by this column when the value was zero. I un-commented this line.
  • In allocate_net_gen_by_gen_esc(), we no longer allow frac_from_g_tbl to be greater than 100%. This was previously happening when the mwh reported in the g table were greater than the mwh reported in the gf table. However, numbers greater than 100% was causing the frac_missing_from_g_tbl to become negative, which was resulting in nonsensical allocations. We implement the same cap on frac_from_bf_tbl in allocate_fuel_by_gen_esc().
  • In allocate_fuel_by_gen_esc(), when calculating frac_cap, the code had been dividing capacity_mw by capacity_mw_unit_fuel. However, this was resulting in some nonsensical allocations because fuel is being allocated by PM-fuel, not by unit. We changed this to divide by capacity_mw_pm_fuel instead. This is consistent with how frac_cap is calculated in the allocate_net_gen_by_gen_esc() function
  • Rename adjust_energy_source_codes() to adjust_msw_energy_source_codes() to more precisely describe what the function does

• Adds new entries to the manual emissions factor tables for NOx and SO2 to represent fuel-boiler combinations that had previously been getting dropped from the data due to this bug.
• Changes the pudl version that we use in our environment from catalyst-cooperative/pudl@main to grgmiller/pudl@oge_release. This will give us more control over performing fixes like this in the future.
• Adds a new validation check to the EIA-923 data cleaning process to verify that for each plant, the total allocated fuel and generation matches the total fuel and generation reported in the input generation and fuel table (basically that the allocation process is not dropping or inflating the data).

Updates plant primary fuel identification

• When assigning the plant primary fuel based on the most consumed fuel, we were previously assigning this based on the fuel with the highest fuel_consumed_mmbtu. However, we should be using fuel_consumed_for_electricity_mmbtu since we want to assign the primary fuel used for electricity generation.
• Sometimes nuclear generators report 0 fuel consumption in EIA-923. Since we were assigning a plant's primary fuel first based on fuel consumption, this meant that sometimes if a nuclear plant had a backup fossil generator, the plant was being assigned the fuel code of that backup generator. To fix this, we now assign the primary fuel of any plant that contains a nuclear unit based on the nameplate capacity of the unit.

Updates to gross to net generation conversions

• Previously, when converting CEMS gross generation to net generation, we had filtered out any ratios that were greater than 1.5 or less than 0.2. However, these values were somewhat arbitrary and turning out to be too wide of a range. For example, when there was a large discrepancy between CEMS gross generation and EIA-923 net generation, we were scaling the CEMS generation to match, even though the fuel consumption and emissions reported in CEMS also disagreed and were not being scaled. This was leading to instances where a plant was using CEMS CO2 totals but EIA-923 net generation totals, resulting in the plant having abnormally high emission rates. To be consistent, if we are going to use CEMS data at all, we want to make sure that the net generation values are reasonable given the reported net generation. After analyzing three years (2019-2021) of annual gross to net ratios, both at the plant and subplant levels, it appears that generally the interquartile range of GTN ratios is between 0.75 and 1.00, with an upper bound around 1.25. Thus, we are now using 0.75 as the lower bound for filtering out ratios, and 1.25 as the upper bound.
• Previously, the backstop gross to net generation approach if all other conversion factors were not available was to assume that gross generation equaled net generation (i.e. a GTN ratio of 1). However, as identified in https://github.com/singularity-energy/open-grid-emissions/issues/177, the EIA has default gross to net conversion factors for each prime mover that they use. This PR introduces these PM-specific gross to net conversion factors as the default backstop option now. As noted in the issue, there are still improvements that need to be made before #177 can be closed, but this is a step in the right direction.

Flags potentially anomalous generated emission factors

• When outputting annual plant level data, we add a new validation check that calculates a generated co2 rate, and flags any rates that appear to be anomalous, so that we can manually inspect these plants to see if there are any unexpected results. For this test, we define anomalous values two ways. On the high end, the check flags the plant if the co2 rate is higher than 15,000 lb/MWh. On the low end, the check flags any plants that have rate lower than 10 lb/MWh but higher than 0 MWh.

Add eGRID2021 to the list of downloads for validation

• This release adds eGRID 2021 data to our list of downloads, and updates the list of non-grid connected plants based on additions to the list in eGRID 2021.

Fixes a bug that was leading to incorrect balancing authority assignments

• Fixes an issue where a plant with no reported ba_code was getting filled with the incorrect code based on the ba_name.
  • We have also identified a new known issue that is not fixed in this release: In comparing our 2021 data to the eGRID 2021 data, we found that there are some plants that EIA-860 identifies as being in ISNE that are getting assigned to NYIS by pudl, and thus are categorized in a different BA than they are in eGRID. All of these plants seem to be physically located in the state of New York, but are listed with an ISNE BA code. Also, all of these plants are pretty small. See: https://github.com/catalyst-cooperative/pudl/issues/2255. We will work to address this with the pudl team.