dynamic params and facets #4

larsbuntemeyer · 2022-10-02T11:21:13Z

This is just adding my first experience with ESGF CORDEX API. It's basically just adapting request parameters and facets for different ESGF projects.

jbusecke

Dear @larsbuntemeyer thank you so much for the engagement here! I really appreciate your help with this.
Your Pr has exposed some issues in my original implementation(especially the lack of tests) and made me think of more general cases (which is very great).
I think that I would like to understand how we can generalize this code to even more mips in a general way (hopefully increasing the usefulness).
I left some detailed comments, but except for some smaller issues (hardcoding table_id=Amo) I think this Pr is a very good addition. Lets iterate on the details and then we can merge this?

jbusecke · 2022-10-05T07:52:53Z

pangeo_forge_esgf/dynamic_kwargs.py

 ) -> Tuple[List[str], Dict[str, Dict[str, str]]]:

-    table_id = facets_from_iid(iid).get("table_id")
+    # really hacky!
+    table_id = "Amon"  # facets_from_iid(iid).get("table_id")


I am not sure this is a good solution. I think basically we need a "facet string" argument that encodes the names of facets as they appear in the instance id.
It would be great to get a sample of more (if there are) different archives (not sure what the right term for the highest level (e.g CMIP, CORDEX) is.
Once we get an idea how many there are and how the different instance ids are formatted we might be able to generalize this more.

Sorry, this was just a quick hack (i didn't actually want to commit that)... I think the highest level term is called project as done, e.g., in the facet search example...

jbusecke · 2022-10-05T07:54:33Z

pangeo_forge_esgf/params.py

@@ -0,0 +1,27 @@
+# dataset id templates
+cmip6_template = "mip_era.activity_id.institution_id.source_id.experiment_id.variant_label.table_id.variable_id.grid_label.version"
+cordex_template = "project.product.domain.institute.driving_model.experiment.ensemble.rcm_name.rcm_version.time_frequency.variable"


Ah this actually partially fits into my earlier comment. But hardcoding "table_id" is still not ideal.
Which facet describes time frequency in CORDEX? I assume its "time_frequency"? Are the values "conpatible" with CMIP Controlled vocabulary?

I dived a little more into the ESGF API documentation and added two functions to retrieve that information from the API now.

Yes, the time frequency in CORDEX is described in "time_frequency", the values are not 100% percent compatible but they contain basically the same vocabulary as in CMIP5, e.g., "day", "mon", "1hr", etc... However, there is no indication of the time cell method, e.g., "1hrPt" as in CMIP6...

I think we could also access the time frequency facet although it's not part of the dataset_id. It still has different names for the ESGF projects, (e.g. time_frequency in CORDEX but frequency in CMIP6). In general, the CORDEX vocabulary is derived from CMIP5 vocabulary. However, this could easily become a class attribute as you suggested...

jbusecke · 2022-10-05T07:59:26Z

test_cordex.py

+
+recipe_inputs = asyncio.run(generate_recipe_inputs_from_iids(iids, ssl=sslcontext))
+print("DONE")
+print(recipe_inputs)


This is very nice! I need to add tests for CMIP cases too.
Can we actually assert here that recipes were generated (and that they have attributes we desire?).
Testing here will be tricky, since we might have to account for certain glitchy things on the ESGF server side. But fir now this is a great step forward!

Just to mention here, that the actual recipe creation won't work with the latest version of pange-forge-recipes because of pangeo-forge/pangeo-forge-recipes#418... when having an SSLObject in the FilePattern...

jbusecke · 2022-10-05T08:02:47Z

pangeo_forge_esgf/recipe_inputs.py

@@ -140,6 +146,9 @@ async def _esgf_api_request(
    if params["latest"] == "true":
        if "version" in facets:
            del facets["version"]


I have previously wondered that this seems yanky. If I set "latest" to false in the requests I can actually provide the version. Is that behavior the same for CORDEX?

I don't really know.. I think in CORDEX, datasets are always the "latest" version while retracted versions are removed from ESGF...

… facets

larsbuntemeyer · 2022-10-05T15:45:55Z

Thanks for your all your comment @jbusecke ! I really appreciate it! I'll clean this up now and let you know when i am ready to merge!

larsbuntemeyer · 2022-10-05T15:50:56Z

pangeo_forge_esgf/utils.py

+    """Requests the dataset_id string template for an ESGF project"""
+    if url is None:
+        url = "https://esgf-node.llnl.gov/esg-search/search"
+    params = {
+        "project": project,
+        "fields": "*",
+        "limit": 1,
+        "format": "application/solr+json"
+    }
+    r = requests.get(url, params)
+    #print(r.status_code)
+    return r.json()["response"]["docs"][0]["dataset_id_template_"][0]
+
+
+def facets_from_template(template: str):
+    """Parse the (dataset_id) string template into a list of (facet) keys"""
+    regex = r"\((.*?)\)"
+    return re.findall(regex, template)


I dived a little bit into the ESGF API documentation and added now these two functions that should be able to retrieve the dataset_id_template_ (and parse it into facets) for any ESGF project. I don't use them yet but i think this is better than what i did in the params module, e.g.,

template = utils.get_dataset_id_template("CMIP6") # returns '%(mip_era)s.%(activity_drs)s.%(institution_id)s.%(source_id)s.%(experiment_id)s.%(member_id)s.%(table_id)s.%(variable_id)s.%(grid_label)s'

which can be parsed into:

utils.facets_from_template(template)

['mip_era', 'activity_drs', 'institution_id', 'source_id', 'experiment_id', 'member_id', 'table_id', 'variable_id', 'grid_label']

hmmm, here we have member_id instead of variant_label, that seems not right, seems that both work...

Oh this was a long standing issue of confusion for me. I have only ever encountered different value for these keys in DCPP experiments. An example here:

Here you can see that variant_label has a different value then member_id:

So long story short I think I mixed those values up! I believe that this occurence here, needs to be changed (which your PR does) since in some cases the value of 'member_id' does not uniquely identify a single dataset. For most experiments the values are interchangeable AFAIK.

jbusecke · 2022-10-06T10:40:37Z

This looks really cool. Ultimately I wonder if we should substantially refactor this logic and make it more object oriented.
Just thinking out loud:
Should we have something like this:

class ESGFBuilder:
    def __init__(project, ):
        ...defines class properties like the facets from project type via the request you outlined above...
    
    ... set up the connection to the server for the duration of the session...
    ...logic that retrieves urls etc...
    ...method that returns recipes...

I will have to invest some more time into this refactor, and wrapping my mind around async, but I just wanted to brain dump this here in case you had some input.

I really like the direction this is going for, generalizing this tool for different ESGF projects!

jbusecke · 2022-10-06T10:51:49Z

README.md

+```
+
+using a different url:


Can you clarify if this is necessary or if this is just to show that the functionality works on all the ESGF search nodes?

Yes, that was just to show, that you can choose a different url. CORDEX searches should actually also work for all ESGF nodes.

jbusecke · 2022-10-06T10:53:12Z

pangeo_forge_esgf/dynamic_kwargs.py

@@ -96,9 +97,11 @@ async def response_data_processing(
    session: aiohttp.ClientSession,
    response_data: List[Dict[str, str]],
    iid: str,
+    ssl: ssl.SSLContext = None,


For now this is great, but adding all these input arguments is a bit of a code smell to me. I think we should aim to set up a 'Connection' or 'Client' object once and then pass this around the different logic that needs to make a request?

jbusecke · 2022-10-06T10:53:53Z

pangeo_forge_esgf/dynamic_kwargs.py

@@ -110,7 +113,7 @@ async def response_data_processing(
    # print(urls)
    print(f"{iid}: Check for netcdf 3 files")
    pattern_kwargs = {}
-    netcdf3_check = await is_netcdf3(session, urls[-1])
+    netcdf3_check = await is_netcdf3(session, urls[-1], ssl)


jbusecke · 2022-10-06T10:55:52Z

pangeo_forge_esgf/recipe_inputs.py

@@ -48,6 +49,8 @@
    "esgf.ichec.ie",
    "esgf.nci.org.au",
    "esgf.rcec.sinica.edu.tw",
+    "esgf1.dkrz.de",


Nice. Thanks

jbusecke · 2022-10-06T10:57:57Z

pangeo_forge_esgf/recipe_inputs.py

                picked_data_node = data_node
                print(f"DEBUG: Picking preferred data_node: {picked_data_node}")
                break
            else:
                print(f"Got status {status} for {matching_data_node['instance_id']}")
        elif len(matching_data_nodes) == 0:
-            print(f'DEBUG: Data node: {data_node} not available')
+            print(f"DEBUG: Data node: {data_node} not available")


There are a lot of these formatting issues here. Do you think it would be useful if I prepend a PR implementing proper pre-commit linting and you can refactor this? Or would that cause more trouble at this point?

#5 would be ready to merge. Let me know if it would help merging that before.
I personally would have a much easier time to focus on the functional changes you made here if we can get rid of the formatting stuff.

Yes, absolutely! I woud very much appreciate the pre-commit config!

Ok let me merge #5 and then you can rebase this.

for more information, see https://pre-commit.ci

Add pre-commit config

…amic_kwargs

…rge-esgf into cordex-requests

larsbuntemeyer · 2022-10-17T22:04:56Z

Just some thoughts here for myself: I have to rethink the mip keyword, it does not make sense. It should be project which is the API keyword. There is an additional complication in the CORDEX project since there is an additional project keyword in the facet search, e.g., there can be project=cordex where the project facet is hardcode in the dataset_id_template:

"id":"cordex.output.EUR-44.MPI-CSC.MPI-M-MPI-ESM LR.historical.r1i1p1.REMO2009.v1.mon.tas.v20150609|esgf1.dkrz.de",
"dataset_id_template_":["cordex.%(product)s.%(domain)s.%(institute)s.%(driving_model)s.%(experiment)s.%(ensemble)s.%(rcm_name)s.%(rcm_version)s.%(time_frequency)s.%(variable)s"]

or project=cordex-reklies where the project facet can be parsed:

"id":"cordex-reklies.output.EUR-11.GERICS.MIROC-MIROC5.historical.r1i1p1.REMO2015.v1.mon.tas.v20170329|esgf1.dkrz.de",
 "dataset_id_template_":["%(project)s.%(product)s.%(domain)s.%(institute)s.%(driving_model)s.%(experiment)s.%(ensemble)s.%(rcm_name)s.%(rcm_version)s.%(time_frequency)s.%(variable)s"]

I have to make sure to search all cordex project not only project=cordex...

larsbuntemeyer · 2022-10-18T18:54:24Z

More thoughts:

If i chose project=CORDEX-Reklies, the product facet gets inactive:

That's really weired since it is still part of the dataset_id.... hmmm.

It works now, but required some ugly exceptions. Have to clean it up some more.

jbusecke · 2022-10-18T20:36:01Z

If i chose project=CORDEX-Reklies, the product facet gets inactive:

Is there a value when you query the API directly? I wonder if this is some glitch in the GUI

Also is there a list of all available projects? I have to admit we are moving into somewhat unfamiliar territory for me here, and I hope that having a full list will give us an idea how to most efficiently generalize this.

larsbuntemeyer · 2022-11-05T14:22:46Z

I'll try to remove some of the if statements that are hacked in depending on the project. I think, this could then be more generalized. The ESGF CORDEX project is not always as consistent as it could be, that's why there might be come exceptions. Sorry for the late responses here, i had to dig through some old grumpy Fortran code the last week... :-/

larsbuntemeyer · 2022-11-07T20:39:31Z

Although the recipe run works locally, feedstock creation is not yet possible due to pangeo-forge/pangeo-forge-recipes#79 and pangeo-forge/pangeo-forge-recipes#418.

jbusecke · 2023-07-28T13:02:20Z

Hey @larsbuntemeyer just checking in here (sorry for the long silence). I just wanted to make you aware of some new developments in PGF-recipes (see #9 (comment)) that in my mind now completely eliminate the need for the dynamic_kwargs.py module. I am planning to fully deprecate this logic and move towards handling those issues in the recipe generation itself.

I have started (and will continue) to work on #15 which reflects these plans. I would be curious if you agree that the upstream changes eliminate your need for the dynamic_kwargs module as well?

larsbuntemeyer · 2023-10-06T12:11:11Z

Hey @jbusecke , thanks for looking back into this after all! Sorry for the long absence, i had to kick some model runs regularly which took all my attention! I agree, that the refactoring makes this redundant. I also have to get myself up to date with the Beam refactor in PGF now.

jbusecke · 2023-12-14T20:27:01Z

Hey @larsbuntemeyer just digging out of a mountain of gh notifications and found this. I have quite substantially refactored this package since the new pangeo-forge version (based on apache beam) became available.

I removed the dynamic_kwargs.py entirely. The process was very imprecise and slow anyways. We can actually now dynamically chunk by providing a custom function to the pangeo-forge-recipe stage. Check out this recent presentation for some details. I have made a new repo specifically for chunking algorithms, so if you had some custom logic in mind, please check it out: https://github.com/jbusecke/dynamic_chunks
I also substantially refactored the ESGF client. This was all really done to serve my CMIP6 ingestion (), which now works really well, but I have to apologize for moving fast and destructive here. I think we could still accomodate CORDEX data here, but I think this would have to be a new PR?

Please let me know if you are still interested in this, or have (understandibly) moved on.

larsbuntemeyer · 2023-12-18T09:13:34Z

Hey @jbusecke thanks for the updates! I have been following a bit your work and just watched your Pangeo showcase recently, great work! I have been moving on a bit with pange-forge-cordex to ingest CORDEX data on AWS.

However, this is all based on pangeo-forge-recipes < 0.10.0, so instead of refactoring that, i will check out your work and see how far i get. But a new PR will definitely make sense, so this one should be closed!

larsbuntemeyer and others added 8 commits October 2, 2022 13:20

added dynamic params and facets for other ESGF projects

0348186

added CORDEX example

f64a938

API update for test_script

e1b6155

examample in test_script

415005c

added esgf1.dkrz.de data node and 308 response

ca1ffc2

added ssl keyword and test_cordex.py script

f2f8513

some format cleaning

a81bd26

hack for CORDEX API

7b2b027

jbusecke requested changes Oct 5, 2022

View reviewed changes

added two functions to retrieve the project's dataset_id_template and…

06397a3

… facets

larsbuntemeyer commented Oct 5, 2022

View reviewed changes

jbusecke reviewed Oct 6, 2022

View reviewed changes

jbusecke and others added 12 commits October 6, 2022 13:01

Add pre-commit config

653d637

[pre-commit.ci] auto fixes from pre-commit.com hooks

ad37153

for more information, see https://pre-commit.ci

Update parsing.py

c18eb2c

[pre-commit.ci] auto fixes from pre-commit.com hooks

66737c0

for more information, see https://pre-commit.ci

Update pyproject.toml

3d00487

Create .flake8

fa0d756

Update dynamic_kwargs.py

dc6f333

Update recipe_inputs.py

904a256

Merge pull request jbusecke#5 from jbusecke/add-pre-commit

7bd23dd

Add pre-commit config

added dynamic params and facets for other ESGF projects

4bcaec9

added CORDEX example

81a6f27

API update for test_script

d7c9196

larsbuntemeyer and others added 9 commits October 17, 2022 23:11

added esgf1.dkrz.de data node and 308 response

0a11650

added ssl keyword and test_cordex.py script

161c89f

some format cleaning

f51dd7a

hack for CORDEX API

b6eb964

format updates

1ec6eee

added frequency to default_params in _esgf_api_request for use in dyn…

b5ba1fa

…amic_kwargs

format fix

16be9ee

Merge branch 'cordex-requests' of github.com:larsbuntemeyer/pangeo-fo…

91644ff

…rge-esgf into cordex-requests

added base_dict to params

50def3b

larsbuntemeyer added 3 commits October 18, 2022 19:47

added utilities for CORDEX project string

b791c4c

updated special facet searches for CORDEX and CORDEX-Reklies

59670f6

update of test script for CORDEX-Reklies

46be3cd

larsbuntemeyer and others added 4 commits November 5, 2022 16:38

default params are now used for api_requests, removed some CORDEX hacks

d27af7b

run pre-commit

c40f55a

fixed type parameter

8676f5d

added test_recipe.py

dcd3a74

larsbuntemeyer marked this pull request as ready for review November 7, 2022 20:39

larsbuntemeyer added 2 commits November 8, 2022 11:43

added CORDEX-Adjust project

f7efa42

added CORDEX-ESD project

28295da

larsbuntemeyer mentioned this pull request Apr 25, 2023

Preparing & uploading data euro-cordex/cordex-aws#3

Closed

jbusecke closed this Jan 18, 2024

		```

		using a different url:

dynamic params and facets #4

dynamic params and facets #4

Conversation

larsbuntemeyer commented Oct 2, 2022 • edited Loading

jbusecke left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

larsbuntemeyer Oct 5, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

larsbuntemeyer commented Oct 5, 2022 • edited Loading

Choose a reason for hiding this comment

larsbuntemeyer Oct 5, 2022 • edited Loading

Choose a reason for hiding this comment

jbusecke Oct 6, 2022 • edited Loading

Choose a reason for hiding this comment

jbusecke commented Oct 6, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

larsbuntemeyer commented Oct 17, 2022 • edited Loading

larsbuntemeyer commented Oct 18, 2022 • edited Loading

jbusecke commented Oct 18, 2022

larsbuntemeyer commented Nov 5, 2022 • edited Loading

larsbuntemeyer commented Nov 7, 2022

jbusecke commented Jul 28, 2023

larsbuntemeyer commented Oct 6, 2023

jbusecke commented Dec 14, 2023

larsbuntemeyer commented Dec 18, 2023

larsbuntemeyer commented Oct 2, 2022 •

edited

Loading

larsbuntemeyer Oct 5, 2022 •

edited

Loading

larsbuntemeyer commented Oct 5, 2022 •

edited

Loading

larsbuntemeyer Oct 5, 2022 •

edited

Loading

jbusecke Oct 6, 2022 •

edited

Loading

larsbuntemeyer commented Oct 17, 2022 •

edited

Loading

larsbuntemeyer commented Oct 18, 2022 •

edited

Loading

larsbuntemeyer commented Nov 5, 2022 •

edited

Loading