Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Eia860m eia923 q1 2025 #4047

Open
wants to merge 14 commits into
base: main
Choose a base branch
from
Open

Eia860m eia923 q1 2025 #4047

wants to merge 14 commits into from

Conversation

aesharpe
Copy link
Member

@aesharpe aesharpe commented Feb 5, 2025

Overview

Closes #4038

What problem does this address?

Adds new monthly data from EIA860m and EIA923

What did you change?

  • Add new DOIs for both data sources
  • Update working partitions for EIA860m (not relevant for 923)
  • Update package_data to accommodate new data and changes to old data
  • Update release notes
  • Update minmax rows test

Documentation

Make sure to update relevant aspects of the documentation.

Tasks

Preview Give feedback

Testing

How did you make sure this worked? How can a reviewer verify this?

  • Materialized assets in dagster
  • Ran fast ETL
  • Ran tests

To-do list

Preview Give feedback

@aesharpe aesharpe self-assigned this Feb 5, 2025
@aesharpe aesharpe added eia923 Anything having to do with EIA Form 923 eia860 Anything having to do with EIA Form 860 data-update When fresh data is integrated into PUDL from quarterly or annual updates labels Feb 5, 2025
@aesharpe aesharpe requested a review from cmgosnell February 7, 2025 20:58
@aesharpe aesharpe marked this pull request as ready for review February 7, 2025 20:58
@@ -1552,6 +1552,7 @@
"rr": "RR",
"pl": "PL",
"rv": "RV",
"RT": "TR", # This is based on a guess. No definitive proof. Culprit is plant_id_eia 3935 in October 2024 in the coalmine table.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried to see what the other rows from the same coalmine listed as the primary transportation method to see if there was a clear fix here; sadly there was not. Entries from older years say "RV" and newer years say "TR". I chose TR because it's "RT" backwards....also it only affects two entires so it's a very minimal issue.

Copy link
Member

@cmgosnell cmgosnell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks pretty good to me!! I left some suggestions on the release notes and on the min/max rows.

Comment on lines +47 to +53
* Run quarterly updates for 2025 Q1 including new EIA860m data from 2024-10 through
2024-12.

EIA 923
~~~~~~~
* Run quarterly updates for 2025 Q1 including new EIA 923 data from 2024-09 through
2024-11.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is totally a nit! but we seem to have almost always said "added X...". I would also find the Q# but with these months confusing as an outside person.

Suggested change
* Run quarterly updates for 2025 Q1 including new EIA860m data from 2024-10 through
2024-12.
EIA 923
~~~~~~~
* Run quarterly updates for 2025 Q1 including new EIA 923 data from 2024-09 through
2024-11.
* Added EIA 860m through December 2024. See :issue:`4038` and pr:`4047`.
EIA 923
~~~~~~~
* Added EIA 923 monthly data through September 2024. See :issue:`4038` and pr:`4047`.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i don't think adding in the pudl plant id mapping will change these diffs, but i think we normally suggest doing that mapping before updating these validation tests. to attempt a lightweight small test after the id's are mapped you could first initialize the glue assets and then initialize one of the more processed tables like gens_eia860

Copy link
Member

@cmgosnell cmgosnell Feb 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also you'll probably want to fully re-run these guys anyway given the new archive. oh i see that you've updated these with the new doi already

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i believe you checked in this ~$ version of this file. if i understand correctly this just means you had the file open. you should save it, close it and then check in the non open version of this file and delete this guy

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
data-update When fresh data is integrated into PUDL from quarterly or annual updates eia860 Anything having to do with EIA Form 860 eia923 Anything having to do with EIA Form 923
Projects
Status: In progress
Development

Successfully merging this pull request may close these issues.

EIA 860/860m/923 Q1 2025 Update
2 participants