MHKiT v0.8.0 #321
Merged
MHKiT v0.8.0 #321
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Use caching by default & reduce testing time
* run hindcast only on change * always run hindcast on changes to master
* allow latest versions from all requirements * add python 3.10 and 3.11 * only test ubuntu on pulls to dev; mac & Windows on pull to master * use actions/upload-artifact@v3; setup-python@v3; download-artifact@v3
* automatic Hs threshold * Revert "automatic Hs threshold" This reverts commit 4477580. * automatic Hs threshold * simplify & include MATLAB example for debugging * fix independent storms * Update mhkit/loads/extreme.py Co-authored-by: Adam Keester <[email protected]> * Update mhkit/loads/extreme.py Co-authored-by: Adam Keester <[email protected]> * Update mhkit/loads/extreme.py Co-authored-by: Adam Keester <[email protected]> * Update mhkit/loads/extreme.py Co-authored-by: Adam Keester <[email protected]> * cleanup * Update mhkit/tests/loads/test_loads.py * Update mhkit/tests/loads/test_loads.py Update threshold test value one more time * Update extreme.py * break out nested function, consolidate scipy imports --------- Co-authored-by: ssolson <[email protected]> Co-authored-by: Adam Keester <[email protected]> Co-authored-by: akeeste <[email protected]>
* Add test for loads.extreme.global_peaks function The loads_extreme.global_peaks function was previously missing a test. The test uses a simple function which can be independently analysed. The results of global_peaks and the independent analysis are then compared. * Introduce upcrossing module Previously there was no general means of performing an upcrossing analysis. The load.extreme.global_peaks function could only calculate peaks. The module provides some common methods but also the ability for the user to define their own function over the zero crossing points. * Implement loads.extreme.global_peaks in terms of upcrossing module With the recent addition of the upcrossing module, we can implement the loads.extreme.global_peaks function using it. * minor linting, module docstring, update parameter validation * fix upcrossing docstring * move upcrossing module to utils * update upcrossing import in example * fix last upcrossing docstring and move test to test/utils * update description in the upcrossing notebook * update import of upcrossing into test_upcrossing * typo in test file * final typo fix --------- Co-authored-by: akeeste <[email protected]>
* remove ci folder * update README installation instructions * remove figures folder * remove pypirc
* add silent kwarg to request_parse_workflow and get_netcdf_variables * describe optional use of silent in the cdip example * fix typo * fix netcdf typo
* loads/extreme convert asserts to errors * loads/general convert asserts to errors * loads/graphics convert asserts to errors * power module convert asserts to errors * loads module, convert try-except validation to errors * utils module, convert asserts to errors * dolfyn module, convert asserts to errors * tidal module, convert asserts to errors * river module, convert asserts to errors * wave hindcast, replace asserts with errors * wave/io module, convert asserts to errors * wave module, convert asserts to errors * rename reserved variable min in plot_directional_spectrum * fix miscellaneous typos * catch new error type * fix test logic and parameter name * minor review comments, fix f strings, fix messages, etc * list correct types in error messages, f strings, etc * standardize error messages for optional parameters * Apply suggestions from ssolson's 2nd review
…278) * check is samples min or max included in the contour half * ensure requested samples are within range of contour values
* Added limits to variable_interpolation and added 3 array input capability to create_points Co-authored-by: Browning <[email protected]> Co-authored-by: ssolson <[email protected]>
* loads/general * loads/extreme conversion to xarray * test and bugfix for loads/general/bin_statistics * fix bug in bin_statistics where std=0 * correct bin_statistics test data * update bin_statistics test * fix dimension name in mler_wave_amp_normalize * formatting fixes * update return types, add optional dimension argument * add Series and DataArray to type check for mler_coefficients * update argument and docstring for time_dimension * update dimension variables to time_dimension * rename time_dimension to frequency_dimension where required
* power/quality add basic formatting * initial conversion or power/quality submodule * add xarray tests for power.quality * fix variable assignment in power.quality * power.characteristics add basic formatting for xr conversion * update error messages * finish converting power.quality to xarray * fix spaces/formating in docstrings * clean up conversion of inputs from pandas to xr.dataset * add tests for xarray * update handling of timestamps in power.characteristics for xr * fix length in power.quality.harmonics * fix length in power.quality.harmonics, again * add frequency_dimension and time_dimension arguments * remove old imports * remove obsolete argument from THCD tests * type check on to_pandas * add type and value checks for time_dimension and frequency_dimension * make grid_freq checks f strings that return the incorrect value * add frequency_dimension valueError * add formal docstring to _convert_to_dataset * add line_to_line type check * restore old naming convention to ac_power_three_phase * update example call to THCD * return hard coded test answers to being recalculated
- black formatting - GitHub actions black - new API key added for NREL's HSDS server - `mhkit.tests.river.test_io` broken out into `mhkit.tests.river.test_io_d3d` and `mhkit.tests.river.test_io_usgs` - added functional tests and error handing tests to `mhkit.tests.dolfyn.test_tools`, `mhkit.tests.mooring.test_mooring` - added error handling tests for `mhkit.tests.river.test_resource`, `mhkit.tests.wave.io.hindcast.test_wind_toolkit`, `mhkit.tests.wave.test_contours`, `mhkit.tests.loads.test_loads`
Skip junk headers in AD2CP files
…o dual_profile
Handle RDI's variable profile shenanigans
* assert to exception * Revert "Ensure interpolation values are within range when sampling contours (#278)" This reverts commit 2387a25. * black format * add a dev requirements * add check for black formatting * add pre commit * change API key * Treat the None as string "None" * black formatting * remove unused imports, add some type checks * no coverage on artifact load * black formatting * add scipy to TEST env * remove no dependencies * remove debug imports * revert to original main.yaml * black * river io to usgs and d3d * move d3d tidal into tidal/io * quick test to call d3d from tidal * snake_case * `product` is deprecated as of NumPy 1.25.0 * fix import path * increase test coverage * black * update test filename * correct name * wait fox James to respond * increase mooring graphics coverage * increase test coverage * increase coverage * catch strings passed as TypeError * Increase coverage on dataTypes * throw error on strings passed * black * remove unused imports * additional input checks * test env contours invalid inputs * uncomment tests * do not include cached API requests in coverage * black * fix & improve type checks * fix bug on slicing after setting min_bins to 4 * increase test coverage * black * black * remove debug * black v24.1.0 * fix implementation from #284 * finalize tests for `cpsd_quasisync_1D` * black * remove "alt_raw" merge artifact * remove positive frequencies test * black formatting * black * black * add back copula tests * pylint: extreme to multiple files * lint * clean up module docstring * fix too many local args * linting * match correct module name * middle of getting linted versions working * mler perfect lint * fix type errors * lint * pylint loads 80% * remove comments * try this * set to 7 * 100% peaks * pylint 10/10 extereme * should run perfect on extreme * mhkit/loads/extreme/ * does this fix install issues? * install package * install * remove [] empyty array input * lint 10/10 * list works; throw error on string data * entire loads module * docstring * 10/10 * type_handling function for checking numeric arrays * typehints * typehints
* convert river.graphics to xarray * convert mhkit.river.resource to xarray * add pandas to xarray helper function * mhkit.river.io.usgs to xarray * convert mhkit.river.io.usgs to xarray * convert river.io.d3d to xarray * fix discharge_to_velocity * move convert_to_dataset into mhkit.utils.data_utils * update dataset labeling * black formatting for previously changed river files * black formatting for data_utils.py * all xarray tests passing * add helper function to convert input to xr.DataArray * update river to use convert_to_dataArray * update utils.convert_to_dataArray * all river modulel tests passing * black formatting * fix typo in river.io.usgs * update tidal.graphics to use utils.convert_to_dataArray * tidal.performance to xarray * remove tidal.performance dead code * convert tidal.resource to xarray * tidal tests passing * add option to return noaa data as xarray, add xarray test * add tidal.performance xarray tests, tests passing * add parameter validation for to_pandas flag * add tests for data_utils functions * black formatting * revert river.io.usgs._read_usgs_json to pandas for now, still allow xarray output * fix cast to xarray in performance test * fix create_points function after simplifying and converting to xarray * black formatting * fix output format for some river resource functions * black formatting * fix final call to exceedance_probability * move convert_to_dataset, convert_to_dataarray to type_handling * replace out variables with descriptive name * correct test function name * correct case on convert_to_dataArray * update return variable name in velocity_profiles * update handling of dataset to dataarray * black formatting
Add capability to read ADCP dual profile configurations
* New TI function, fix noise bug in psd * Update notebook * Fix test file * Fix default
* Skip junk data * That's backwards * Improve winriver filesize guess * Updated log file * Black formatting * Cleanup * Fix debugger test * Yet another riverpro bugfix * Cleanup * Clean up ugly code * Code improvement * More cleanup and return statements
* temperature data is signed * Set pitch roll to signed * heading is unsigned * Fix ad2cp temp test * Fix awac pitch/roll data
* lint & hint
* fix type error message in power.quality * convert wave.performance to xarray * remove extraneous variables from test_performance * wave/resource 90 percent converted to xarray * speed up conversion to dataset * wave.resource conversion complete * allow xarray input to wave.contours functions * allow xarray in wave.graphics * minor cleanup * update type handling in wave_number and depth_regime * update error messages in convert_to_dataarray * test_wave_metrics passing * correct typo in graphics * wave tests passing except frequency binned surface_elevation calls * fix bug/memory error with surface_elevation frequency bins * black formatting * allow xarray output in wave.io.cdip * allow xarray output from wave.io.wecsim * make utils function to convert nesed dicts of dataframes to xarray * allow xarray input/output in wave.io.swan * allow xarray input/output in wave.io.ndbc * add xarray test for test_wecsim.py * add xarray test in test_cdip * update dataframe to dataarray case in utils.type_handling * add xarray test to test_ndbc * add xarray tests to test_swan * black formatting * fix single value dataframe to dataarray conversion * black formatting * update docstring in type_handling * update description of to_pandas flag when returning a dict of pd or xr * update ndbc docstrings * move TODO from comment to an issue * update error message * update output types for wavelength * add frequency_dimension and time_dimension kwargs * add check for shape of phases and S in surface_elevation * black formatting * pylint suggestion in power.characteristics * fix cdip example * allow xarray output in wave.io.hindcast functions
* restore correct default for pandas/xarray output * update as_xarray flag to to_pandas in hindcast tests
This PR addresses 2 issues with the hindcast tests: * On PRs tests were only being run on changes to hindcast test files. - Fixed to run on changes to hindcast test and module files. * The prepare cache job was failing due to 6 hour job limit. - Fixed by breaking out each hindcast cache test into its own job. * Additionally this PR updates action packages to the latest version where possible.
conda-incubator/setup-miniconda@v3 to install conda.
- Uses python version correctly - Installs MHKiT dependencies in conda build - Use coverage actions instead of coverage CLI - Uses an updated coverage version with lcov support - Adds an `environment.yaml` file for conda environment install - Special case for MacOS-latest (macos 14) and Py 3.8 - CI test job `set-os` logic modified to now run all OS on push to `develop`
- `hindcast-calls` job now uses the environment.yaml file & forces the correct coverage version as added in #317 - Wind cache waits on wave cache completion
Closed
|
I pushed to develop in #320. After the job link below passes the wind hindcast (~6 hours) I will merge the develop branch into the v0.8.0 branch and everything should pass and be ready to go. https://github.com/MHKiT-Software/MHKiT-Python/actions/runs/8972281680 |
* hindcast must all skip or succeed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
MHKiT v0.8.0
We're excited to announce the release of MHKiT v0.8.0, which brings a host of new features, enhancements, and bug fixes across various modules, ensuring compatibility with Python 3.10 and 3.11, and introducing full xarray support for more flexible data handling. Significant updates in the Wave and DOLfYN modules improve functionality and extend capabilities.
Python 3.10 & 3.11 Support
MHKiT now supports python 3.10 and 3.11. Support for 3.12 will follow in the next minor update.
Wave Module
Enhancements:
Automatic Threshold Calculation for Peaks-Over-Threshold: We've introduced a new feature that automatically calculates the "best" threshold for identifying significant wave events. This method, originally developed by Neary, V. S., et al. in their 2020 study, has now been translated from Matlab to Python, enhancing our existing peaks-over-threshold functionality.
Wave Heights Analysis: A new function,
wave_heights, has been added to extract the heights of individual waves from a time series. This function uses zero up-crossing analysis to accurately measure wave heights, improving upon our previous methods which only provided the maximum value between up-crossings.Enhanced Zero Crossing Analysis: Building on the above, the zero crossing code previously embedded in
global_peakshas been isolated into a helper function. This modular approach not only refines the codebase but also supports new functionalities such as calculating wave heights, zero crossing periods, and identifying crests.Bug Fixes:
Contour Sampling Error in Wave Contours: A bug identified in
mhkit.wave.contours.samples_contourhas been resolved. The issue occurred when period samples defined using the maximum period resulted in values outside the interpolation range of the contour data. This was corrected by ensuring that all sampling points are within the interpolation range and adjusting the contour data selection process accordingly.Xarray Support
MHKiT functions now fully support the use of xarray for passing and returning data.
DOLfYN
Thanks to the many user contributions and users of MHKiT the DOLFYN module include a significant number of enhancements and bug fixes.
Enhancements:
Altimeter Support: Enhanced the Nortek Signature Reader to add capability for reading ADCP dual profile configurations.
Data Handling Improvements: Introduced logic to skip messy header data that can accumulate during measurements collected via Nortek software on PCs and Macs.
Instrument Noise Subtraction: Added a function to subtract instrument noise from turbulence intensity estimation using RMS calculations, providing results that differ by approximately 1% from the existing standard deviation-based "I" property.
Improved File Handling: Updates for RDI files to handle changing "number of cells" and variable "cell sizes," which are now bin-averaged into the largest cell size.
Bug Fixes:
Power Spectra Calculation: Fixed a bug where a given noise value was not being subtracted from the power spectra, and noise was inadvertently added as an input to dissipation rate calculations.
Improved Header Handling: Allowed RDI reader to skip junk headers effectively.
Nortek Reader C Types Update: Adjusted C types in the Nortek reader to handle below-zero water temperatures and to allow pitch and roll values to go negative.
River & Tidal: D3D
Added limits to
variable_interpolationand added 3 array input capability tocreate_pointsDeveloper Experience
Black formatting
Black formatting is now enforced on all MHKiT files. This ensures consistent formatting across the MHKiT package.
Linting & Type Hints
MHKiT is in the process of enforcing pylint and adding type hints to all functions. Currently this has been achieved and is enforced in the Loads and Power modules.
CI/CD
This release introduces significant reduction in testing time for development. This is achieved by reducing the number of tests for pulls against the develop branch and only running hindcast test when changes are made to it. A bug in the hindcast CI was fixed which only ran on changes to the hindcast tests instead of the hindcast module. Additionally the wave and wind hindcast needed to be separated in 2 jobs due to the excessive time taken to run a wind cache. This created a number of follow on PRs around solidifying the logic of these job. A special case for Python 3.8, pip, and Mac OS was added to use homebrew to install NetCDF and HDF5 to get tests to pass.
develop#306Clean Up
MHKiT fixed an implementation error where functions used assert instead of built in errors for type and value checking. Additionally these PRs removed unused files, fixed typos, and created an argument which allows users to run CDIP API calls silently.
This PR replaced #307