Release v0.8.3

Bugfix Release 0.8.3

Summary

This release of ripple1d fixes several bugs that were identified in a large production run. Those bugs were detailed and summarized in #268.

Bug Fixes

• When a new terrain is downloaded but contains only nodata values, NullTerrainError is raised, and the endpoint process will fail.
• When a new terrain is downloaded but the HEC-RAS CreateTerrain subprocess fails silently, the output is checked, and a RasTerrainFailure error is raised. Additionally, bad_projection_test.py has been added to test that the error will raise.
• When a source model is conflated such that the most downstream XS for a NWM reach a has higher river station than the most upstream reach, a BadConflation error is raised.
• When sub model extraction is attempted on a reach that would have one or zero cross-sections but is not tagged as eclipsed, a SingleXSModel error is raised. The endpoint response was also cleaned up such that eclipsed reaches no longer return a path to a nonexistent 123.ripple1d.json. Logging has been improved for all these cases.
• flow indexing in the create_flow_depth_combinations function has been improved to avoid rounding values. A test has been added for one of the models that was causing problems.
• Handling of interpolated cross-sections has been improved such that they may have sub models created, may be run, and may be used in the create FIM lib steps. A new test model -- winooski -- has been added with interpolated cross-sections.
• Null (blank) downstream reach values are now loaded as 0. This is consistent with how HEC-RAS handles them. The new winooski model contains null downstream reach length at the most downstream cross-section.

Closes #281
Closes #282
Closes #291
Closes #292
Closes #295
Closes #298

Release v0.8.1

Re-release 0.8.1

Important

Version 0.8.0 was released without having incorporated bugfixes added after v0.8rc1. This update adds in the necessary bugfixes to reflect the most up-to-date version of the software. The text below is a copy of the 0.8.0 release notes.

Summary

This release adds a suite of diagnostic metrics on model quality and terrain agreement to the ripple1d outputs. Model quality metrics reflect how well a HEC-RAS model conforms to standards of practice for developing HEC-RAS models and provide easy access to many model attributes that would be checked during an engineering model QC. Terrain agreement metrics quantify how well the terrain generated in the create_ras_terrain endpoint matches the source model terrain. Access to these metrics give ripple1d users actionable information to inform which HEC-RAS models should be preferred when models overlap. Furthermore, elevation-specific terrain agreement metrics can be used to alert users when a FIM product substantially disagrees with a source model geometry (indicative of lower FIM quality).

Additional features have been added to handle HEC-RAS warnings about hydraulic table start points and cross-section naming.

Features Added

Hydraulic Table Start Elevation

When a hydraulic table for a cross-section has a minimum elevation below the section invert, HEC-RAS generates a pop up warning as demonstrated in #249. In the warning, the software specifies that any subsequent saves to the geometry file will automatically adjust the hydraulic table minimum elevation to be above the section invert. To mimic this functionality, a new function call has been added to the RasGeomText constructor. The call will check for any occurrences of the htab issue, adjust the minimum elevation, and overwrite the geometry file with the updated value.

HEC-RAS implements some convoluted process to update both the htab start value and the increments. We elected to simply update the htab start elevation to the cross-section invert and preserve the original engineer-determined increments.

Sub Model Cross-Section Names

In earlier versions of ripple1d, the software attempted to carry cross-section names from source model to sub model. When that naming would have resulted in decreasing values in the upstream direction, the stationing was adjusted to preserve increasing values in the names. The renaming process led to a couple of bugs in downstream processes (ex #225 and #266). Since cross-section names do not affect hydraulic computation, the software will now rename all sub model cross-sections to an autoincremented value (ex 1, 2, 3, etc).

Additionally, the extract_submodel endpoint has been refactored to fix a potential bug in the junction walking step. For cases where more than one junction separated the u/s and d/s cross-sections or cases where the junctions were not reported from upstream to downstream in the HEC-RAS geometry file, the extract_submodel endpoint would only grab a single intermediate reach. No examples of this behavior have been found in production, but the bug was tested in development with a testing HEC-RAS model.

Terrain Agreement Metrics

A new step has been added to the create_ras_terrain endpoint that will generate a json of terrain agreement metrics. The json is saved to a file with suffix ".terrain_agreement.json" that is saved to the sub model directory. The JSON contains a suite of metrics that have been documented in the Technical Summary of the readthedocs page (https://ripple1d.readthedocs.io/en/latest/tech_summary.html) and the create_ras_terrain endpoint page (https://ripple1d.readthedocs.io/en/latest/endpoints/create_ras_terrain.html).

Model Quality Metrics

A new step has been added to the extract source model endpoint which extracts source model information to determine the quality of the source model. These metrics are populated in the cross section layer located in the source model geopackage layer for every cross section. The submodel will inherit these metrics during the subset gpkg phase.

Bug Fixes

• A feature was added in v0.7.0 to allow html rendering of the jobs endpoint, however, the endpoint was not updated to accept a flag toggling between json and html responses. Additionally, an oversight in the project manifest meant that the html and css templates for the jobs page were not transferred to ripple1d when installed via pip. This has been resolved so that the jobs endpoint returns a json response by default, and an html response will be returned when the key-value pair f=html is in the query string.
• readthedocs was failing to build our documentation on their server. The docs folder has been updated such that it will build via their service. Additionally, the changelog will autogenerate now. Some old windows dependencies and imports have been removed since they are not used to run HEC-RAS any more and needed unnecessary logic to avoid breaking the readthedocs build service.

Closes #118
Closes #207
Closes #208
Closes #209
Closes #225
Closes #249
Closes #266
Closes #267
Closes #268
Closes #269
Closes #270

Release v0.8.0

Feature Release 0.8.0

Summary

This release adds a suite of diagnostic metrics on model quality and terrain agreement to the ripple1d outputs. Model quality metrics reflect how well a HEC-RAS model conforms to standards of practice for developing HEC-RAS models and provide easy access to many model attributes that would be checked during an engineering model QC. Terrain agreement metrics quantify how well the terrain generated in the create_ras_terrain endpoint matches the source model terrain. Access to these metrics give ripple1d users actionable information to inform which HEC-RAS models should be preferred when models overlap. Furthermore, elevation-specific terrain agreement metrics can be used to alert users when a FIM product substantially disagrees with a source model geometry (indicative of lower FIM quality).

Additional features have been added to handle HEC-RAS warnings about hydraulic table start points and cross-section naming.

Features Added

Hydraulic Table Start Elevation

When a hydraulic table for a cross-section has a minimum elevation below the section invert, HEC-RAS generates a pop up warning as demonstrated in #249. In the warning, the software specifies that any subsequent saves to the geometry file will automatically adjust the hydraulic table minimum elevation to be above the section invert. To mimic this functionality, a new function call has been added to the RasGeomText constructor. The call will check for any occurrences of the htab issue, adjust the minimum elevation, and overwrite the geometry file with the updated value.

HEC-RAS implements some convoluted process to update both the htab start value and the increments. We elected to simply update the htab start elevation to the cross-section invert and preserve the original engineer-determined increments.

Sub Model Cross-Section Names

In earlier versions of ripple1d, the software attempted to carry cross-section names from source model to sub model. When that naming would have resulted in decreasing values in the upstream direction, the stationing was adjusted to preserve increasing values in the names. The renaming process led to a couple of bugs in downstream processes (ex #225 and #266). Since cross-section names do not affect hydraulic computation, the software will now rename all sub model cross-sections to an autoincremented value (ex 1, 2, 3, etc).

Additionally, the extract_submodel endpoint has been refactored to fix a potential bug in the junction walking step. For cases where more than one junction separated the u/s and d/s cross-sections or cases where the junctions were not reported from upstream to downstream in the HEC-RAS geometry file, the extract_submodel endpoint would only grab a single intermediate reach. No examples of this behavior have been found in production, but the bug was tested in development with a testing HEC-RAS model.

Terrain Agreement Metrics

A new step has been added to the create_ras_terrain endpoint that will generate a json of terrain agreement metrics. The json is saved to a file with suffix ".terrain_agreement.json" that is saved to the sub model directory. The JSON contains a suite of metrics that have been documented in the Technical Summary of the readthedocs page (https://ripple1d.readthedocs.io/en/latest/tech_summary.html) and the create_ras_terrain endpoint page (https://ripple1d.readthedocs.io/en/latest/endpoints/create_ras_terrain.html).

Model Quality Metrics

A new step has been added to the extract source model endpoint which extracts source model information to determine the quality of the source model. These metrics are populated in the cross section layer located in the source model geopackage layer for every cross section. The submodel will inherit these metrics during the subset gpkg phase.

Bug Fixes

• A feature was added in v0.7.0 to allow html rendering of the jobs endpoint, however, the endpoint was not updated to accept a flag toggling between json and html responses. Additionally, an oversight in the project manifest meant that the html and css templates for the jobs page were not transferred to ripple1d when installed via pip. This has been resolved so that the jobs endpoint returns a json response by default, and an html response will be returned when the key-value pair f=html is in the query string.
• readthedocs was failing to build our documentation on their server. The docs folder has been updated such that it will build via their service. Additionally, the changelog will autogenerate now. Some old windows dependencies and imports have been removed since they are not used to run HEC-RAS any more and needed unnecessary logic to avoid breaking the readthedocs build service.

Closes #118
Closes #207
Closes #208
Closes #209
Closes #225
Closes #249
Closes #266
Closes #267
Closes #268
Closes #269
Closes #270

Release v0.8rc1

Feature Release 0.8.0

Summary

This release adds a suite of diagnostic metrics on model quality and terrain agreement to the ripple1d outputs. Model quality metrics reflect how well a HEC-RAS model conforms to standards of practice for developing HEC-RAS models and provide easy access to many model attributes that would be checked during an engineering model QC. Terrain agreement metrics quantify how well the terrain generated in the create_ras_terrain endpoint matches the source model terrain. Access to these metrics give ripple1d users actionable information to inform which HEC-RAS models should be preferred when models overlap. Furthermore, elevation-specific terrain agreement metrics can be used to alert users when a FIM product substantially disagrees with a source model geometry (indicative of lower FIM quality).

Additional features have been added to handle HEC-RAS warnings about hydraulic table start points and cross-section naming.

Features Added

Hydraulic Table Start Elevation

When a hydraulic table for a cross-section has a minimum elevation below the section invert, HEC-RAS generates a pop up warning as demonstrated in #249. In the warning, the software specifies that any subsequent saves to the geometry file will automatically adjust the hydraulic table minimum elevation to be above the section invert. To mimic this functionality, a new function call has been added to the RasGeomText constructor. The call will check for any occurrences of the htab issue, adjust the minimum elevation, and overwrite the geometry file with the updated value.

HEC-RAS implements some convoluted process to update both the htab start value and the increments. We elected to simply update the htab start elevation to the cross-section invert and preserve the original engineer-determined increments.

Sub Model Cross-Section Names

In earlier versions of ripple1d, the software attempted to carry cross-section names from source model to sub model. When that naming would have resulted in decreasing values in the upstream direction, the stationing was adjusted to preserve increasing values in the names. The renaming process led to a couple of bugs in downstream processes (ex #225 and #266). Since cross-section names do not affect hydraulic computation, the software will now rename all sub model cross-sections to an autoincremented value (ex 1, 2, 3, etc).

Additionally, the extract_submodel endpoint has been refactored to fix a potential bug in the junction walking step. For cases where more than one junction separated the u/s and d/s cross-sections or cases where the junctions were not reported from upstream to downstream in the HEC-RAS geometry file, the extract_submodel endpoint would only grab a single intermediate reach. No examples of this behavior have been found in production, but the bug was tested in development with a testing HEC-RAS model.

Terrain Agreement Metrics

A new step has been added to the create_ras_terrain endpoint that will generate a json of terrain agreement metrics. The json is saved to a file with suffix ".terrain_agreement.json" that is saved to the sub model directory. The JSON contains a suite of metrics that have been documented in the Technical Summary of the readthedocs page (https://ripple1d.readthedocs.io/en/latest/tech_summary.html) and the create_ras_terrain endpoint page (https://ripple1d.readthedocs.io/en/latest/endpoints/create_ras_terrain.html).

Model Quality Metrics

A new step has been added to the extract source model endpoint which extracts source model information to determine the quality of the source model. These metrics are populated in the cross section layer located in the source model geopackage layer for every cross section. The submodel will inherit these metrics during the subset gpkg phase.

Bug Fixes

• A feature was added in v0.7.0 to allow html rendering of the jobs endpoint, however, the endpoint was not updated to accept a flag toggling between json and html responses. Additionally, an oversight in the project manifest meant that the html and css templates for the jobs page were not transferred to ripple1d when installed via pip. This has been resolved so that the jobs endpoint returns a json response by default, and an html response will be returned when the key-value pair f=html is in the query string.
• readthedocs was failing to build our documentation on their server. The docs folder has been updated such that it will build via their service. Additionally, the changelog will autogenerate now. Some old windows dependencies and imports have been removed since they are not used to run HEC-RAS any more and needed unnecessary logic to avoid breaking the readthedocs build service.

Closes #118
Closes #207
Closes #208
Closes #209
Closes #225
Closes #249
Closes #266
Closes #267
Closes #268
Closes #269
Closes #270

Release v0.7.0

Feature Release 0.7.0

Users Changelog

This release of ripple1d attempts to speed up the known water surface elevation (kwse) computes by parallelizing the creation of the depth grids. To do this, all kwse runs are ran without producing depth grids. From these initial kwse runs the rating curve database are created and used to inform the boundary condition for the next reach upstream. Once all of the boundary conditions are known, a second kwse run is ran to produce the depth grids in parallel. To make this happen some changes were necessary to the endpoints.

In addition to adding/modifying the endpoints, this release of ripple1d makes significant updates to both ripple1d execution and logging. This includes running each endpoint as a separate subprocess which allows the user to have the ability to dismiss running jobs. This should be handy for when jobs appear to have hung up. Dismissing these hung up jobs will free up cpu for new jobs.

Features Added

Write depth grids argument

A new argument boolean "write_depth_grids" was added to the "run_incremental_normal_depth" and "run_known_wse" endpoints. This allows the user to specify whether ripple1d should compute raw RAS depth grids or not.

Create rating curves database

A new endpoint called "create_rating_curves_db" was added. This endpoint is a post processing step that creates a rating curve database from RAS plan hdf results. This endpoint only requires 2 args: "submodel_directory" and "plans. The location of the rating curve database is inferred from the submodel directory. It will be located directly in the submodel directory and will be named as the network reach name; e.g., "2820002/2820002.db".

The "create_rating_curves_db" endpoint checks for the presence of the depth grid for each profile and records its existence or absence along with the plan suffix in the database table. The columns are "plan_suffix" and "map_exist" and are of type string and boolean, respectively.

Create FIM library update

The "create_fim_lib" endpoint no longer produces the rating curve database.

Update to the Ripple1d workflow

The user should take care with the args when calling the new/modified endpoints; specifically the plan names. The recommended order for calling these endpoints is:

1. run_known_wse : with "write_depth_grids" set to false and "plan_suffix" set to ["ikwse"]
2. create_rating_curves_db: with "plans" set to ["ikwse"]
3. run_known_wse: with "write_depth_grids" set to true and "plan_suffix" set to ["nd","kwse"]
4. create_rating_curves_db: with "plans" set to ["nd","kwse"]
5. create_fim_lib: with "plans" set to ["nd","kwse"]

Subprocess encapsulation

The execution of all endpoints are now encapsulated within subprocesses

Logs and database

• Huey and flask logs have been combined into a single log file (server-logs.jsonld).
• Huey.db has been renamed to jobs.db
• The process id for each endpoint are now tracked in a p_id field of the task_status table in jobs.db
• A huey_status field has been added to the task_status table. This field tracks the execution status of the endpoint subprocess.
• A new table called task_logs has been added to jobs.db. This table contains stdout, stderr, and results stemming from endpoint subprocesses.
• A proof of concept graphical (html-based) job status page has been added

Closes #187
Closes #188
Closes #237
Closes #238
Closes #239
Closes #240

Release v0.6.3

Bugfix Release 0.6.3

Users Changelog

This release of ripple1d fixes several bugs identified during testing.

Bug Fixes

• Technical Documentation has been updated with high level summary of package functionality.
• The ID column was removed from geopackage layers. All code dependencies on the ID column have been removed.
• Now only reaches that are connected via the "to_id" are considered eclipsed reaches.
• Precision has been added to the rating curves used to inform the incremental normal depth runs by reducing the amount of rounding.
• CRS is now stored as WKT instead of EPSG in the ripple.json file to more robustly represent all possible CRSs; e.g., ESRI.

Closes #59
Closes #133
Closes #192
Closes #219
Closes #221

Release v0.6.2

Bugfix Release 0.6.2

This Release fixes several bugs associated with conflation. To aid in identifying and fixing these bugs, improvements were made in the logging for the conflation endpoint. In summary, the fixes and changes incorporated in this PR improve the consistency and quality of the conflation process, computations, and metrics in genera with special attention for handling junctions and headwater reaches.

Release v0.6.1

Bugfix Release 0.6.1

Users Changelog

This release of ripple1d fixes several bugs identified during testing.

Features Added

• A minor change was added to the logging behavior to improve error tracking.

Bug Fixes

• A bug causing increasing processing time when calling creat_ras_terrain in parallel mode.

• A bug in the extract_submodel endpoint which failed when trying to grab the upstream cross section. A check was added for the eclipsed parameter, where if true no geopackage will be created.

• Several bugs associated with the create_fim_lib endpoint:

  1. The library_directory arg was not being implemented correctly in the function.
  2. If a fim.db already exists append fuctionality has been implemented.
  3. If the directory containing the raw RAS depth grids is empty the clean up function will not be invoked.

• Resolves issues introduced when a concave hull from a source model where cross section existed in the wrong direction (resulting in a multipart polygon). A check was added to correct direction and reverses the cross section if it was drawn incorrectly. This is limited to the development of the concave hull and does not modify the cross section direction for use in the modeling.

Closes #178
Closes #190
Closes #193
Closes #189

Release v0.6.0

Feature Release 0.6.0

Users Changelog

This release of ripple1d adds 2 args to the create_fim_lib endpoint, adds a concave hull of the cross sections to the geopackage, and fixes a bug associated with the depth grids.

Features Added

New library directory argument

A new required arg, "library_directory", has been added to the create_fim_lib endpoint. This new arg specifies where the output directory for the FIM grids and database.

New cleanup argument

A new required arg, "cleanup", has been added to the create_fim_lib endpoint. If this arg is True the raw HEC-RAS depth grids are deleted. If False they are not deleted.

Concave hull of cross sections

A layer representing the concave hull of the cross sections has been added to the geopackage for the source model and the network based model. It also improves how the concave hull is handled at junctions by explicitly creating a junction concave hull and then merging it in with the xs concave hull.

Bug Fixes

• An error was arising when all normal depth runs resulted in water surface elevations that were below the mapping terrain which means no resulting depth grids were being produced. Previously the code assumed at least 1 depth grid would be present. This has been fixed by obtaining the "terrain part" of the raw RAS grid from the RAS terrain instead of the first depth grid in the raw RAS result folder.

Developers Changelog

• Feature/concave hulls by @mdeshotel in #175
• Feature/delete raw ras grids by @mdeshotel in #179
• Bug/depth grids by @mdeshotel in #180
• Feature/fim directory by @mdeshotel in #182

Closes #151
Closes #166
Closes #169
Closes #170

Release v0.5.0

Feature Release 0.5.0

Users Changelog

This release of ripple1d incorporates geometry extraction, conflation, and conflation metrics into the API, and fixes several bugs.

Features Added

Conflation improvements

• The source HEC-RAS river centerline is now clipped to the most upstream and most downstream cross sections prior to starting conflation. This helps prevent identifying network reaches that are far away from the cross sections and improves the accuracy of the conflation.
• Overlapped reaches are now tracked and documented in the conflation json file
• A bbox column has been added to the network parquet file for faster reading. This was especially needed for the new conflation endpoint since each request needs to load the parquet file. Load times without the bbox column were between 5-20 seconds; this is reduced to 1-2 seconds witht the bbox column.
• The conflation function now reads locally instead of from s3.
• The conflation function no longer creates a STAC item.
• RAS metadata is now added to the conflation json.
• The source network's metadata is now added to the conflation json.
• Length and flow units are now documented in the conflation json file.

Conflation Metrics

Three metrics are computed to asses the qualitiy of the conflation.

• Coverage: The the start and end location of the reach coverage is computed as a ratio of the length of the network reach.
• Lengths: The lengths between the most upstream cross section and most downstream cross section along the network reach and source HEC-RAS Model's centerline is computed. The ratio of the two lengths is also provided.
• XS: The distance between where the network reach and HEC-RAS Model's centerline intersects the cross sections is computed. A similar comparison is performed using the cross section's thalweg location and the network reaches intersection location with the cross sections. The mean, min, max, std, and quartiles are provided as a summary for both comparisons.

Geometry Extraction improvements

• A new function to verify .prj file is a HEC-RAS file has been added.
• The extracted geopackage now contians a non-spatial metadata table for the souce HEC-RAS model.
• Tests have been added for extracting geopackage from HEC-RAS model.
• Additional attributes are added to the source model gpkg for downstream use.
• Units are extracted from the source RAS model and added to metadata.

API

• An endpoint was added for extracting geometry and relevant metdata for the soure HEC-RAS models and storing it in in a geopackage.
• An endpoint to compute conflation metric was added.
• An endpoint for conflation (which includes metrics calculations) was added.
• Tests were added for the conflation, conflation metrics, and geopackage endpoints.

Bug Fixes

• Reaches whose conflation results indicate upstream and downstream cross sections are the same are now considered a failed conflation.
• The function to create a concave hull for the cross sections has been improved when junctions are present.
• Eclipsed reaches are now better identified and are documented in the conflation json with a boolean.
• A check is now performed to ensure cross sections intersect the source HEC-RAS model's river centerline. If cross sections do not intersect the centerline they are dropped.
• A conflation json is no longer written for source HEC-RAS models that fail to conflate.
• Handling has been added to subset gpkg endpoint for river stationings of interpolated. These river stations contain an "*" to indicate interpolated cross section.
• Several issues with the automated API tests were identified and fixed.
• API tests no longer re-run gpkg_from_ras and conflate_model for every reach; just once per source test model.
• When API tests pass the resulting files are now removed automatically. Resulting files for tests that fail are not removed so that the tester can better trouble shoot.

Developers Changelog

• Update to incorporate v0.4.1 changes by @slawler in #144
• Merge main to dev after 0.4.2 release by @ar-siddiqui in #146
• Feature/src gpkg endpoint by @slawler in #155
• Feature/conflation metric by @slawler in #154
• Bugfix/extract submodel by @mdeshotel in #159
• Bugfix/conflation tests by @mdeshotel in #158
• Bugfix/stationing handle river stationing of interpolated cross sections by @mdeshotel in #162
• Bugfix/conflation by @mdeshotel in #161
• Bugfix/compute metrics by @mdeshotel in #164

Closes #92
Closes #117
Closes #137
Closes #147
Closes #148
Closes #149
Closes #150
Closes #160