Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add Ocean Gliders format output option to seaxexplorer processing #212

Closed
wants to merge 2 commits into from

Conversation

callumrollo
Copy link
Collaborator

This PR is under development and is waiting on several currently active proposed edits to the Ocean Gliders format before it can be finalised

This will resolve #21

Copy link

codecov bot commented Dec 12, 2024

Codecov Report

Attention: Patch coverage is 13.20755% with 46 lines in your changes missing coverage. Please review.

Project coverage is 53.77%. Comparing base (7ec1b8e) to head (778a6b9).
Report is 7 commits behind head on main.

Files with missing lines Patch % Lines
pyglider/utils.py 4.87% 39 Missing ⚠️
pyglider/seaexplorer.py 41.66% 7 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #212      +/-   ##
==========================================
- Coverage   55.11%   53.77%   -1.35%     
==========================================
  Files           9        9              
  Lines        1700     1750      +50     
==========================================
+ Hits          937      941       +4     
- Misses        763      809      +46     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@jklymak
Copy link
Member

jklymak commented Dec 12, 2024

A quick skim - I'm not clear that a lot of this cannot be handled via the yaml.

For the rest, it would be nice to keep new variables like platform id in the yaml as well. I've not looked but perhaps that requires a new variable type that just accepts a constant value.

Maybe we should have a quick call, but I would rather see a general approach rather than special "og" carve outs

outname = outdir + id0 + fnamesuffix + '.nc'
_log.info('writing %s', outname)
ds.to_netcdf(outname, 'w',
encoding={'time': {'units': 'seconds since 1970-01-01T00:00:00Z',
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For example, 'time'/'TIME' could be replaced by a time-encoder yaml entry. eg calls to ds.time could be replaced by ds[time_var_name] or similar?

Not sure about 'latitude' etc, but why do they need to be translated back and forth? For distance over ground? Maybe just use inside that function?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can localise lon/lat to just change within those functions, I'll make a new PR with ds[time_var_name] and see what it looks like

f"sea{ds.attrs['glider_serial'].zfill(3)}",
attrs={"long_name": "glider serial number"},
)
ds["DEPLOYMENT_TIME"] = np.nanmin(ds.TIME.values)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The yamls have a "conversion" field for each variable that tries to fine the method in utils.py. So this could look like

netcdf_variables:
    DEPLOYMENT_TIME:
         "long_name": "date of deployment"
         ...
         "conversion": "nanmin_time"

with the accompanying change in utils.py. We may want to think of a way to make that more general (np.nan_min, "DEPLOYMENT_TIME) if yaml accepts tuples.

@VeckoTheGecko
Copy link
Contributor

VeckoTheGecko commented Dec 17, 2024

6651931 (via #210) introduced a bunch of formatting changes that will likely cause merge conflicts. Here are merge and rebase workflows to avoid these conflicts (it depends on having pre-commit installed - see the contributing docs).

I recommend first creating a temp branch off of your branch of interest to test these workflows and make sure the output is as expected. I have tested these and they seem to work well.

# merge all the way up to the commit that introduced the formatting changes
git merge 66519318^1

# merge the commit that introduced the formatting changes, prioritising the working branch during conflicts
git merge -s ours 66519318 --no-commit 
pre-commit run --all-files
git commit --no-edit

# merge the rest of the way
git merge main

or by doing a rebase (if you want a clean history)

# Rebase up to the commit introducing formatting changes
git rebase 66519318^1

# Rebase onto the commit introducing formatting changes.
# Accept working branch changes (i.e., "theirs" during rebase
# - different to merge. See git rebase docs) during conflicts.
# During the rebase we will run the formatter after applying
# changes from commits, before committing so that everything is compliant.
git rebase -X theirs -i 66519318
# Replace all "pick" with "edit" or "e". Start the rebase...

# Repeatedly run
git add -u && pre-commit run --all-files && git rebase --continue
# to run pre-commit tooling, stage modified files, and continue
# until rebase is complete. Sometimes pre-commit can't autofix
# issues, then you need to fix them manually.
# If you make a mistake you can always do `git rebase --abort`

# Finally, rebase onto main
git rebase main

@callumrollo
Copy link
Collaborator Author

I'll try a different approach following Jody's suggestion of a new yaml entry that makes empty variables/variables with one value

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Output OG 1.0
3 participants