Add support for daily output frequency #9

duncanwp · 2022-10-14T15:59:14Z

I needed to support the daily table_id for my climatebench-feedstock so have made a first pass at fixing #3 here.

Happy for any feedback / other approaches.

for more information, see https://pre-commit.ci

jbusecke

Dear @duncanwp, thank you very much for this very helpful contribution. I think the changes are all good as is.
I think it would be a good idea to add some simple unit tests for get_timesteps_simple to ensure we get consistent results for a set of given inputs.

jbusecke · 2022-10-17T18:02:23Z

pangeo_forge_esgf/dynamic_kwargs.py

@@ -13,21 +13,35 @@
    # TODO: I might not want to allow this in the ocean and ice fields. Lets see
 )

+daily_divisors = sorted(
+    [1, 30, 360, 365, 3653]
+    # TODO: It  might be more efficient to default to the underlying file chunks?


I dont think so. Dealing with the underlying chunks, and outputting them in a way that makes sense to the user is one of the core tasks pangeo-forge aims to solve. These divisons should rather reflect scientifically useful units to e.g. avoid splitting e.g. a yearly chunk with a few days left, as this would make certain operations (annual average) much slower to compute.
These look good for now, but I was wondering about different calendars and how we could maybe influence which divisors to use depending on the calendar of the data.
Just thinking out loud here.

Yeah, that makes sense. As far as I know there isn't any ESGF metadata to describe the calendar though and we'd have to open the files to inspect the time coordinate metadata directly which would be pain.

Yeah I think this is a problem to be pushed to later work.

jbusecke · 2022-10-17T18:03:45Z

pangeo_forge_esgf/dynamic_kwargs.py

 allowed_divisors = {
    "Omon": monthly_divisors,
    "SImon": monthly_divisors,
    "Amon": monthly_divisors,
+    "day": daily_divisors,


Would these dividers also work generally for other daily data (see here: https://github.com/WCRP-CMIP/CMIP6_CVs/blob/master/CMIP6_table_id.json)?

I honestly have no idea, I didn't realise there were different day table_id and can't find any documentation on what they represent. They might be the same but it seems a bit redundant. Happy to include them or not depending on how much you want to protect against potential future issues.

I will try to check this PR out in the next few days, and maybe run some tests with the 'other' daily frequencies.

jbusecke · 2022-10-17T18:06:20Z

pangeo_forge_esgf/dynamic_kwargs.py

+    elif "day" in table_id:
+        timesteps = [
+            m * 30 for m in months
+        ]  # This is close enough without worrying about calendars etc.


See my comment above. I guess dealing with calendars is a whole can of worms, but I am thinking it might be worth to put some of these thoughts into a dedicated issue to keep track of it? Again for now this seems fine for me!

Sure, that sounds like a plan

duncanwp · 2022-11-04T15:11:54Z

Great, happy to add a test or two - but there doesn't seem to be a framework. How do you want to set it up? Would it make sense as a separate PR?

jbusecke

I think we can merge this soon. I hope to get some time to play around with this in the next days. @duncanwp could you provide me with the instance ids of the data you are using for climatebench right now?

pangeo_forge_esgf/dynamic_kwargs.py

jbusecke · 2023-01-23T16:56:57Z

pangeo_forge_esgf/dynamic_kwargs.py

@@ -13,21 +13,35 @@
    # TODO: I might not want to allow this in the ocean and ice fields. Lets see
 )

+daily_divisors = sorted(
+    [1, 30, 360, 365, 3653]
+    # TODO: It  might be more efficient to default to the underlying file chunks?


Yeah I think this is a problem to be pushed to later work.

jbusecke · 2023-01-23T16:57:52Z

pangeo_forge_esgf/dynamic_kwargs.py

 allowed_divisors = {
    "Omon": monthly_divisors,
    "SImon": monthly_divisors,
    "Amon": monthly_divisors,
+    "day": daily_divisors,


I will try to check this PR out in the next few days, and maybe run some tests with the 'other' daily frequencies.

Co-authored-by: Julius Busecke <[email protected]>

for more information, see https://pre-commit.ci

duncanwp · 2023-01-23T21:19:09Z

I don't have an explicit list sorry but I extract all NorESM2-LM files for all ensemble members matching the following experiments and variables:

experiments = [
               '1pctCO2', 'abrupt-4xCO2', 'historical', 'piControl', # CMIP
               'hist-GHG', 'hist-aer', # DAMIP
               'ssp126', 'ssp245', 'ssp370', 'ssp370-lowNTCF', 'ssp585' #	ScenarioMIP
]
variables = [
             'tas', 'tasmin', 'tasmax', 'pr'
]

See the original (hacky!) script here: https://github.com/duncanwp/ClimateBench/blob/main/prepare_data.py

duncanwp · 2023-02-02T19:15:06Z

bump

duncanwp · 2023-04-10T21:23:01Z

bump

jbusecke · 2023-07-28T12:54:45Z

Hi @duncanwp, sorry for the radio silence here. I am just now returning to this work in my context of transforming CMIP data from ESGF. I think that some work I have started upstream in pangeo-forge-recipes could entirely eliminate the need for dynamic kwargs generation in pangeo-forge-esgf. This would be really nice since the issue is solved on a much more general level. The new beam-refactor enables a lot more flexibility.
If you are ok with this I am tending towards moving this discussion/feature further upstream?

duncanwp · 2023-08-02T00:25:25Z

No worries, sounds good. I'd still be keen to look at daily ClimateBench variables as a potential use-case so would appreciate a pointer on where to get started with that!

jbusecke · 2023-08-02T10:11:03Z

I am also still very curious about that use case @duncanwp, and I truly appreciate your patience. I think we can try to support that from the LEAP data-ingestion side (or at least focus discussions there).
Could you raise an issue here so we can discuss this usecase with respect to the new beam refactor (hopefully entraining @cisaacstern in the process 😄).

FYI here is some more info on LEAP and how we are trying to handle data ingestion (even though your use case is more of a derived data product than a canonical ingestion): https://leap-stc.github.io/leap-pangeo/jupyterhub.html#i-have-a-dataset-and-want-to-work-with-it-on-the-hub-how-do-i-upload-it

Either way having an outline of your plan there would be very helpful to keep things focused.

duncanwp · 2023-08-03T00:51:40Z

Sure, sounds good! I just created an issue over there: leap-stc/data-management#43

jbusecke · 2023-08-07T11:37:06Z

Fantastic. Thank you so much.

duncanwp and others added 5 commits October 14, 2022 16:54

Add support for daily output frequency

18964ea

[pre-commit.ci] auto fixes from pre-commit.com hooks

a3f6873

for more information, see https://pre-commit.ci

Fix exception style

1391ba7

Revert unintended change

625a911

[pre-commit.ci] auto fixes from pre-commit.com hooks

7fbe763

for more information, see https://pre-commit.ci

jbusecke requested changes Oct 17, 2022

View reviewed changes

duncanwp mentioned this pull request Nov 4, 2022

Determine time chunks based on calendar #11

Closed

duncanwp mentioned this pull request Jan 6, 2023

Proposed Recipes for ClimateBench pangeo-forge/staged-recipes#243

Closed

jbusecke reviewed Jan 23, 2023

View reviewed changes

duncanwp and others added 2 commits January 23, 2023 21:14

Update pangeo_forge_esgf/dynamic_kwargs.py

2ba4f64

Co-authored-by: Julius Busecke <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

49035f0

for more information, see https://pre-commit.ci

jbusecke mentioned this pull request Jul 28, 2023

dynamic params and facets #4

Closed

duncanwp mentioned this pull request Aug 3, 2023

ClimateBench dataset leap-stc/data-management#43

Open

duncanwp closed this Aug 3, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for daily output frequency #9

Add support for daily output frequency #9

duncanwp commented Oct 14, 2022

jbusecke left a comment

jbusecke Oct 17, 2022

duncanwp Nov 4, 2022

jbusecke Jan 23, 2023

jbusecke Oct 17, 2022

duncanwp Nov 4, 2022

jbusecke Jan 23, 2023

jbusecke Oct 17, 2022

duncanwp Nov 4, 2022

duncanwp commented Nov 4, 2022

jbusecke left a comment

jbusecke Jan 23, 2023

jbusecke Jan 23, 2023

duncanwp commented Jan 23, 2023

duncanwp commented Feb 2, 2023

duncanwp commented Apr 10, 2023

jbusecke commented Jul 28, 2023

duncanwp commented Aug 2, 2023

jbusecke commented Aug 2, 2023

duncanwp commented Aug 3, 2023

jbusecke commented Aug 7, 2023

Add support for daily output frequency #9

Add support for daily output frequency #9

Conversation

duncanwp commented Oct 14, 2022

jbusecke left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

duncanwp commented Nov 4, 2022

jbusecke left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

duncanwp commented Jan 23, 2023

duncanwp commented Feb 2, 2023

duncanwp commented Apr 10, 2023

jbusecke commented Jul 28, 2023

duncanwp commented Aug 2, 2023

jbusecke commented Aug 2, 2023

duncanwp commented Aug 3, 2023

jbusecke commented Aug 7, 2023