Remove outliers and impute missing values in EIA 930 hourly demand

[zaneselvans]

Description

Apply the same outlier / error detection heuristics and correlated time series imputation methods that we currently use on the FERC-714 hourly demand data to produce a complete and plausible version of the EIA 930 hourly demand data.

Motivation

• In our processing of the FERC 714 hourly demand data we use some outlier / error detection heuristics originally developed by @truggles for application to the EIA 930 hourly demand data.
• In the spring of 2024 we integrated the EIA 930 data, but are currently only publishing a relatively untouched version of it, since folks using it for resource adequacy want to be able to do their own outlier detection (as outliers drive RA requirements / failures).
• Now @awongel and others are interested in having a cleaned / imputed version of the EIA 930 for other modeling purposes.
• Catalyst has resources from NSF POSE to support open source contributors, and this seems like a great opportunity for collaboration, and incremental improvement of PUDL, that will create a resource of lasting value to a wider base of users.

Scope

In Scope

• Apply Tyler Ruggles' heuristics to the EIA-930 data to identify outliers and errors, and null them out.
• Run the correlated time series imputation code that we currently apply to the FERC 714 hourly demand data on all 3 of the hourly demand / production time series that come from the EIA 930, producing 3 new tables derived from:
  • core_eia930__hourly_net_generation_by_energy_source
  • core_eia930__hourly_operations
  • core_eia930__hourly_subregion_demand
• The correlated time series imputation seems like it won't be applicable to the hourly interchange data, since it's not necessarily going to be correlated with anything else -- it could vary totally independent of actual demand. Also, we know it doesn't reconcile the generation / demand differences well everywhere, and the modifications from the time series imputation should be pretty minor, so we might not want to touch it at all.
• Ideally, we should end up with some generic code that can be used on both the EIA 930 and the FERC 714, with minimal duplication. The trick seems like it'll be organizing the EIA 930 data appropriately to be processed by the existing code (which is already somewhat modular and should be reusable... but we'll see!)

Out of Scope

• This issue is not related to applying @jdechalendar's reconciliation code to the EIA 930 data. We still might want to do that later, but I think it would build upon the outputs being generated here.

Tasks

Preview Give feedback

1. Use FERC 714 cleaning & imputation functions in state_demand as template for EIA 930
2. Pull out reusable helper functions that apply to both FERC 714 and EIA 930 cleaning & imputation
3. Define new EIA 930 assets analogous to _out_ferc714__hourly_imputed_demand
4. Cleaned core_eia930__hourly_net_generation_by_energy_source
5. Cleaned core_eia930__hourly_operations
6. Cleaned core_eia930__hourly_subregion_demand
7. Plot pre- and post-imputation data with synthetic outliers to sanity check the outputs
8. Define some summary and/or statistical comparisons to see how we've changed the data
9. Create resource definitions for the new EIA 930 (parquet only) outputs
10. Look at how cleaning / imputation affects relationship to interchange data

[zaneselvans]

• The out_ferc714__hourly_planning_area_demand table that we produce has not been cleaned up.
• All the time series cleaning work is currently done in the preparation of the out_ferc714__hourly_estimated_state_demand output table.
• So we don't currently have an exact analog for the cleaned EIA 930 timeseries -- at least not that's being written out and distributed.
• The ephemeral _out_ferc714__hourly_imputed_demand asset looks like the closest thing.
• So the high level machinery that's doing the nulling and imputation is currently FERC 714 specific, but is making use of lower level functions from the timeseries_cleaning.py module, and the FERC 714 functions are pretty short, so if there are any differences between the FERC 714 and EIA 930 data (like the non-demand columns that need to be preserved) maybe it makes sense to create analogous functions that are EIA 930 specific.