Liveocean #179

rsignell-usgs · 2022-08-31T14:23:27Z

redo of #154

pangeo-forge-bot · 2022-08-31T14:24:49Z

It looks like there may be a problem with the structure of your PR.

I encountered a FilesChangedOutsideRecipesSubdirError('This PR changes files outside the recipes/ directory.').

pangeo-forge-bot · 2022-08-31T15:06:48Z

It looks like there may be a problem with the structure of your PR.

I encountered a FilesChangedOutsideRecipesSubdirError('This PR changes files outside the recipes/ directory.').

pangeo-forge-bot · 2022-08-31T15:21:02Z

It looks like there may be a problem with the structure of your PR.

I encountered a FilesChangedOutsideRecipesSubdirError('This PR changes files outside the recipes/ directory.').

rsignell-usgs · 2022-08-31T15:25:59Z

@rabernat and @cisaacstern, the pre-commit.ci failed, but because of other recipes, not my "liveocean" one, right?

I also fixed a few of the issues with the pre-commit, and the formatting issues with some other recipes which were autofixed by:

 pre-commit run --all-files

I believe at least the liveocean recipe should be good to go!

cisaacstern · 2022-08-31T15:42:43Z

👋 @rsignell-usgs thanks for re-opening this. Re: pre-commit, yes this is long-running issue being tracked in #88, and which I plan to resolve soon. You are correct it doesn't affect this recipe. (@andersy005, I've had some recent ideas of how we'll handle this pre-commit issue, will follow up with you separately within the next few days.)

Re: running liveocean, copying a few points I noted #154 (comment) here (didn't realize that PR was closed when I commented there 🤦 ):

... IIUC, this depends on pangeo-forge/pangeo-forge-recipes#399, which has not been released yet. So a couple last blockers (all on my side) before we can run this here:

Release pangeo-forge-recipes. This is easy, can happen anytime.

Our current @pangeo-forge-bot backend is difficult to update with new pangeo-forge-recipes releases. I'm pushing to deploy a new version which will be easier to update this week.

Once the new backend is deployed, I'll need to update https://github.com/pangeo-data/pangeo-docker-images/tree/master/forge to get the latest pangeo-forge-recipes there.

Getting close here!

rsignell-usgs · 2022-08-31T15:45:34Z

Okay @cisaacstern , thanks for the update! 🤞

rsignell-usgs · 2022-08-31T15:58:00Z

BTW, the PR got closed because I deleted my fork (forgetting the PR was still open). I had to revive it with this approach: https://stackoverflow.com/a/73549088/2005869.

But then even though when I reopened the old PR, Github saw the new fork, it was confused, so I closed the PR again and created a new one. 😖

cisaacstern · 2022-09-09T01:04:36Z

Removing the blocked:recipes-release tag, because necessary changes were just released in https://github.com/pangeo-forge/pangeo-forge-recipes/releases/tag/0.9.1. Two of the three issues listed above as blockers here are now complete:

Release pangeo-forge-recipes. This is easy, can happen anytime.
Our current @pangeo-forge-bot backend is difficult to update with new pangeo-forge-recipes releases. I'm pushing to deploy a new version which will be easier to update this week.
Once the new backend is deployed, I'll need to update https://github.com/pangeo-data/pangeo-docker-images/tree/master/forge to get the latest pangeo-forge-recipes there.

rsignell-usgs · 2022-10-20T18:54:30Z

@cisaacstern , any updates here?
(my birthday is Oct 25, and I already have cake lined up....) 😄

rabernat · 2022-10-20T18:59:50Z

Charles is currently on parental leave. 👶 In the meantime, maybe @andersy005 might be able to help move this forward?

andersy005 · 2022-10-20T19:15:48Z

Charles is currently on parental leave. 👶 In the meantime, maybe @andersy005 might be able to help move this forward?

i'm happy to look into this sometime this afternoon...

for more information, see https://pre-commit.ci

andersy005 · 2022-10-20T22:22:25Z

/run liveocean

pangeo-forge · 2022-10-20T22:28:00Z

🎉 The test run of liveocean at 1e8a4db succeeded!

import xarray as xr

store = "https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/test/pangeo-forge/staged-recipes/recipe-run-1279/liveocean.zarr"
ds = xr.open_dataset(store, engine='zarr', chunks={})
ds

andersy005 · 2022-10-20T22:37:40Z

tada The test run of liveocean at 1e8a4db succeeded!

import xarray as xr

store = "https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/test/pangeo-forge/staged-recipes/recipe-run-1279/liveocean.zarr"
ds = xr.open_dataset(store, engine='zarr', chunks={})
ds

@rsignell-usgs, something seems to have gone wrong because the written store appears to be empty. i don't know why :(

....
File ~/mambaforge/envs/playground/lib/python3.10/site-packages/xarray/backends/zarr.py:401, in ZarrStore.open_group(cls, store, mode, synchronizer, group, consolidated, consolidate_on_close, chunk_store, storage_options, append_dim, write_region, safe_chunks, stacklevel)
    385     except KeyError:
    386         warnings.warn(
    387             "Failed to open Zarr store with consolidated metadata, "
    388             "falling back to try reading non-consolidated metadata. "
   (...)
    399             stacklevel=stacklevel,
    400         )
--> 401         zarr_group = zarr.open_group(store, **open_kwargs)
    402 elif consolidated:
    403     # TODO: an option to pass the metadata_key keyword
    404     zarr_group = zarr.open_consolidated(store, **open_kwargs)

File ~/mambaforge/envs/playground/lib/python3.10/site-packages/zarr/hierarchy.py:1347, in open_group(store, mode, cache_attrs, synchronizer, path, chunk_store, storage_options, zarr_version)
   1345         if contains_array(store, path=path):
   1346             raise ContainsArrayError(path)
-> 1347         raise GroupNotFoundError(path)
   1349 elif mode == 'w':
   1350     init_group(store, overwrite=True, path=path, chunk_store=chunk_store)

GroupNotFoundError: group not found at path ''

andersy005 · 2022-11-01T17:05:15Z

@rsignell-usgs, something seems to have gone wrong because the written store appears to be empty. i don't know why :(

@rsignell-usgs, Just realized this is using kerchunk, and everything seems to be working fine when using the reference filesytem

ref_url = "https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/test/pangeo-forge/staged-recipes/recipe-run-1279/liveocean.zarr/reference.json"

ds = xr.open_dataset("reference://", engine='zarr', 
                     backend_kwargs={'consolidated': False, 
                                     'storage_options': {'fo': ref_url, 'remote_options': {'anon': True}, 'remote_protocol': 's3'}}, 
                     chunks={})
ds

We need to update the pangeo-bot and the front-end to remove hard-coded assumption about all datasets being zarr native and ensure the right code snippets are used for different types of recipes supported by pangeo-forge.

rsignell-usgs · 2022-11-14T14:51:10Z

@andersy005 is there anything for me to do at this point, or just hang tight on my end?

rsignell-usgs · 2022-11-28T19:37:40Z

/run liveocean

pangeo-forge · 2022-11-28T19:44:38Z

🎉 The test run of liveocean at 1e8a4db succeeded!

import xarray as xr

store = "https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/test/pangeo-forge/staged-recipes/recipe-run-1279/liveocean.zarr"
ds = xr.open_dataset(store, engine='zarr', chunks={})
ds

rsignell-usgs · 2022-11-28T21:56:31Z

oh, I see. The recipe is working correctly. It's just the code snippet that isn't working.

This notebook works!

rsignell-usgs · 2022-11-28T21:57:14Z

I guess now this can be merged, right?

rsignell-usgs · 2022-11-30T22:01:37Z

@cisaacstern

cisaacstern · 2022-12-01T16:36:08Z

Apologies for the delay, @rsignell-usgs! Thanks for the nudge. Yes, if test run passed then we can merge, which I'll do now.

cisaacstern added blocked:runner-image No current version of runner image is available with the required dependencies. blocked:recipes-release Requires a pangeo-forge-recipes release to run. labels Sep 8, 2022

cisaacstern removed the blocked:recipes-release Requires a pangeo-forge-recipes release to run. label Sep 9, 2022

rsignell-usgs and others added 15 commits October 20, 2022 16:13

adding liveocean recipe

003655b

modifying provider info for Parker MacCready

8dc38ad

Update meta.yaml

47abcc9

downgrade recipes version for cloud environment

f64b592

bump back to 0.9.0 for dataflow

51d1216

adding liveocean recipe

6cb73cd

downgrade recipes version for cloud environment

e608879

bump back to 0.9.0 for dataflow

2c060c1

Update .pre-commit-config.yaml

cafcc1a

[pre-commit.ci] auto fixes from pre-commit.com hooks

9716a14

for more information, see https://pre-commit.ci

pre-commit checked

0878a49

removed extra line

701f4d3

[pre-commit.ci] auto fixes from pre-commit.com hooks

cbf9b05

for more information, see https://pre-commit.ci

Merge main into liveocean

bb65c32

reset setup.cfg

1e8a4db

cisaacstern merged commit 7962f7b into pangeo-forge:master Dec 1, 2022

cisaacstern mentioned this pull request Dec 1, 2022

Add opener code snippet for kerchunk dataset test runs pangeo-forge/pangeo-forge-orchestrator#198

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Liveocean #179

Liveocean #179

rsignell-usgs commented Aug 31, 2022

pangeo-forge-bot commented Aug 31, 2022

pangeo-forge-bot commented Aug 31, 2022

pangeo-forge-bot commented Aug 31, 2022

rsignell-usgs commented Aug 31, 2022 •

edited

Loading

cisaacstern commented Aug 31, 2022

rsignell-usgs commented Aug 31, 2022

rsignell-usgs commented Aug 31, 2022 •

edited

Loading

cisaacstern commented Sep 9, 2022

rsignell-usgs commented Oct 20, 2022 •

edited

Loading

rabernat commented Oct 20, 2022

andersy005 commented Oct 20, 2022

andersy005 commented Oct 20, 2022

pangeo-forge bot commented Oct 20, 2022

andersy005 commented Oct 20, 2022 •

edited

Loading

andersy005 commented Nov 1, 2022

rsignell-usgs commented Nov 14, 2022

rsignell-usgs commented Nov 28, 2022 •

edited

Loading

pangeo-forge bot commented Nov 28, 2022

rsignell-usgs commented Nov 28, 2022 •

edited

Loading

rsignell-usgs commented Nov 28, 2022 •

edited

Loading

rsignell-usgs commented Nov 30, 2022

cisaacstern commented Dec 1, 2022

Liveocean #179

Liveocean #179

Conversation

rsignell-usgs commented Aug 31, 2022

pangeo-forge-bot commented Aug 31, 2022

pangeo-forge-bot commented Aug 31, 2022

pangeo-forge-bot commented Aug 31, 2022

rsignell-usgs commented Aug 31, 2022 • edited Loading

cisaacstern commented Aug 31, 2022

rsignell-usgs commented Aug 31, 2022

rsignell-usgs commented Aug 31, 2022 • edited Loading

cisaacstern commented Sep 9, 2022

rsignell-usgs commented Oct 20, 2022 • edited Loading

rabernat commented Oct 20, 2022

andersy005 commented Oct 20, 2022

andersy005 commented Oct 20, 2022

pangeo-forge bot commented Oct 20, 2022

andersy005 commented Oct 20, 2022 • edited Loading

andersy005 commented Nov 1, 2022

rsignell-usgs commented Nov 14, 2022

rsignell-usgs commented Nov 28, 2022 • edited Loading

pangeo-forge bot commented Nov 28, 2022

rsignell-usgs commented Nov 28, 2022 • edited Loading

rsignell-usgs commented Nov 28, 2022 • edited Loading

rsignell-usgs commented Nov 30, 2022

cisaacstern commented Dec 1, 2022

rsignell-usgs commented Aug 31, 2022 •

edited

Loading

rsignell-usgs commented Aug 31, 2022 •

edited

Loading

rsignell-usgs commented Oct 20, 2022 •

edited

Loading

andersy005 commented Oct 20, 2022 •

edited

Loading

rsignell-usgs commented Nov 28, 2022 •

edited

Loading

rsignell-usgs commented Nov 28, 2022 •

edited

Loading

rsignell-usgs commented Nov 28, 2022 •

edited

Loading