Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

API to create source file and data for OWC software #142

Merged
merged 43 commits into from
Jan 14, 2022
Merged

API to create source file and data for OWC software #142

merged 43 commits into from
Jan 14, 2022

Conversation

gmaze
Copy link
Member

@gmaze gmaze commented Nov 4, 2021 •
edited
Loading

@gmaze gmaze added forQCexpert Argo QC expertise is required enhancement New feature or request argo-core About core variables (P, T, S) labels Nov 4, 2021
@gmaze gmaze self-assigned this Nov 4, 2021
@gmaze gmaze added this to the Go from alpha to beta milestone Nov 4, 2021
gmaze added 7 commits November 4, 2021 16:26
@gmaze
1st day commit
61a27ed
@gmaze
Merge branch 'master' into mat-owc
8da0133
@gmaze
Update xarray.py
889e589 
- filter_qc can take a list of QC fields, instead of using all available
@gmaze
Update xarray.py
acecc78 
Add new filter_scalib_pres method to select pressure, temperature and salinity according to OWC preprocessing requirements
@gmaze
Update xarray
d834bc1 
Add the subsample_pressure method to select vertical levels to keep max 1 level every 10db, starting from the surface (0db)
Add misc utilities functions
@gmaze
Add create_float_source in xarray.py
b0c7b21
@gmaze
Flake8 on xarray new stuff
cf21761
@gmaze gmaze mentioned this pull request Nov 8, 2021
Closed
gmaze added 3 commits November 9, 2021 08:20
@gmaze
Update xarray.py
f2ad9d6 
More dev on create_float_source, add logger
@gmaze
Update xarray.py
b4e41bb
@gmaze
Update xarray.py
bc571c0 
New _where method to preserve Argo data type after xarray where !
Misc changes  to use new argo._where (some need for cast_types removed)
More debug logs
@gmaze gmaze mentioned this pull request Nov 10, 2021
Closed
8 tasks
@gmaze
Copy link
Member Author

gmaze commented Nov 10, 2021

Examples for the validation are given here: #141 (comment)

gmaze added 11 commits November 10, 2021 10:59
@gmaze
Merge branch 'master' into mat-owc
9932cd0
@gmaze
Update xarray.py
ef2ac6a 
New N_PROF, N_LEVELS, N_POINTS accessor properties
@gmaze
Add new step to re-align pressure levels
b1adcb6 
Add the ds_align_pressure method
@gmaze
Update utilities.py
8f34b60 
Redesigned year fraction computation functions to use only pandas   timestamp and timedelta. These new functions are robust to leap years and give precisions down to the second.
@gmaze
Update xarray.py
7044aa4 
- Fix bug in _where method not returning a new object
- Improved __repr__ to include N profiles and levels
@gmaze
Update xarray.py
a09f989
@gmaze
Update xarray.py
a7d08cf 
fix bug where the Matlab output was not  from the pressure re-aligned dataset
@gmaze
Minor stuff
9e96722 
Make sure to output longitude between 0-360
@gmaze
Fill in documentation
0ae15b1 
- Add new section to the documentation (Data quality control) for the new methods
- Modify the create_float_source to be able to handle dataset with more than one float ! Output file name are generated automatically, only the path/folder name can be provided
@gmaze
Start dev of CI tests for new methods
7555e45
@gmaze
New groupby_remap, align_std_bins and complete redesign of create_flo…
3e0b2e5 
…at_source

- Now use a xarray apply_ufunc to bin and align measurements (groupby_remap). This a new accessor method (align_std_bins) to be used even without create_float_source
- Results are similar and obtained faster
@gmaze gmaze mentioned this pull request Nov 26, 2021
Open
@gmaze
Copy link
Member Author

gmaze commented Nov 26, 2021

Contemplating at fixing #155 also

@gmaze gmaze requested review from quai20 and kamwal November 26, 2021 19:23
@kamwal
Copy link

kamwal commented Nov 29, 2021

Hi @gmaze, Thank you for pinning me.
I will look at/test your core around tomorrow if that's fine.

@kamwal
Copy link

kamwal commented Nov 30, 2021

I have started some reviewing process of this code.
Initially the installation and the first run of the code went without any problems.
I need to continue the further code checks (comparisons with matlab code output from the create_float_source) and better understand all the steps undertaken in the code.

@gmaze, are you expecting from me to test the code or rather peer review. I don't have much experience in peer review and it can take a wile for me now.

@gmaze
Copy link
Member Author

gmaze commented Dec 1, 2021

@kamwal
Thanks for looking at this
The review I expect, and sorry I should have been more precise earlier, is:

  • check that you can make the API to run as expected (never know what you get if not ran from my own dev env, despite all CI)
  • perform some comparisons with the Matlab output, I've made a lot, but never sure enough, please take into consideration that the Matlab routine is not perfect and misses some points at the bottom of the profile, which I think we don't miss. See Method to create float_source/<WMO>.mat file for OWC #141 discussion

Poking you with a review, is also a nice way to simply let you know what we've done in here !

@gmaze
Copy link
Member Author

gmaze commented Dec 1, 2021

@kamwal , looking at the code details is clearly not expected, please review as an "end user" rather "code dev" :

thanks !

@gmaze
Copy link
Member Author

gmaze commented Jan 14, 2022

Thanks @quai20 and @kamwal for your review !

@gmaze gmaze merged commit b5dba6d into master Jan 14, 2022
@gmaze gmaze deleted the mat-owc branch January 18, 2022 13:20
@gmaze gmaze mentioned this pull request Mar 4, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kamwal kamwal kamwal approved these changes

@quai20 quai20 quai20 approved these changes

Assignees

@gmaze gmaze

Labels
argo-core About core variables (P, T, S) enhancement New feature or request forQCexpert Argo QC expertise is required
Projects
None yet
Milestone
Development roadmap
Development

Successfully merging this pull request may close these issues.

Method to create float_source/<WMO>.mat file for OWC
3 participants
@gmaze @kamwal @quai20