Add Flow-Only Gauges to rfc_max_stage #312
Comments
MonicaStone-NOAA
added
enhancement
|
@MonicaStone-NOAA do you have an example LID of one of these flow-only gauges? |
|
@CoreyKrewson-NOAA Here's an example of a gauge where thresholds are defined by flow, not stage: https://water.weather.gov/ahps2/hydrograph.php?wfo=sew&gage=tolw1 |
|
awesome. Thanks! |
|
@TylerSchrag-NOAA Did you say that you have a task in mind to convert the wrds api handler to sql? I am working on this and debating if I should just do that now or just update the python code and convert later. Any thoughts? |
|
I am hitting a couple snags on this, mostly because of threshold values in WRDS. The WRDS API converts stage and streamflow behind the scenes when it can. So if I query a location that gives a stage forecast and I ask the API for streamflow forecasts, it will convert the stage to a streamflow and provide that forecast. Because there is no way to ask for gauges that are providing a pure streamflow forecast, I need to query the API for both stage and streamflow forecasts and basically combine them both but removing any duplicate sites, favoring the ones that came from the stage forecasts. In theory this should work but there are lots of examples popping up where this is flawed because of the threshold values. For example, Here is a forecast from TACN4 The stage forecast is below action and thus when we query the API for all locations at action and above, we do not get this site. If we check the API, then we can see that the thresholds are correct. When I query for all streamflow forecast above action, the API will convert the stage to a streamflow and then compare it to the streamflow thresholds that I assume they are also converting. In many cases the stage and streamflow thresholds do not match up. If we look at this same site but for streamflow thresholds we see the following In this case, the API is saying that the action threshold is 720 CFS but it should actually be 1,400 CFS. Because of this the converted streamflow forecast is above 720 and is thus returned by the streamflow query for all locations at action and above. Then when I combine the stage and streamflow forecasts above action, it adds these sites where incorrect action streamflow forecasts exist @GautamSood-NOAA any insights into this dilemna between stage and streamflow forecasts? |
|
Looking at TACN4, I do see the differences between the official flow thresholds and our calculated flow thresholds. Given the notable differences in flow vs calculated flow values, I suspect the WFO used a different rating curve (or different stage values) from what we now have available to us (NRLDB and USGS Rating Depot rating curves). The RFC Forecast API only uses the official thresholds (not the calculated thresholds) which is why that particular location is only at action level. Here are the threshold values from our Location API
|
|
ok good to know about the forecast API using official thresholds. Those calculated flow thresholds looks pretty good. What rating curve is being used for that? |
|
If the particular LID is crosswalked to a USGS Gage, we actually use both (NWS and USGS) rating curves. You'll see that TACN4 has 2 sets of calculated flows. Otherwise, we just use the NWS rating curves |
CoreyKrewson-NOAA
removed this from the
WPOD Q2 New Service/Service Enhancement Requests milestone
This PR, if merged, will replace the existing "rfc_max_forecast" service with a new version of the service whose backend data is now produced by direct access to the WRDS RFC Forecast database as opposed to the WRDS API. As a result: * The `every_five_minute` EventBridge rule that triggered the `viz-wrds-api-handler` Lambda has been modified to instead trigger the `initialize-viz-pipeline` Lambda directly. The viz-wrds-api-handler code is now deprecated and can be eventually removed. * The `initialize-viz-pipeline` Lambda code had to be modified to allow the "rfc" pipeline to be kicked off without referencing any input files - as there are none. This allows it to act on more of a "cron" basis and less of a "file-driven-event" basis. * The new `products/rfc/rfc_max_forecast.sql` query relies upon a few new database structures: * Two new ENUM types - `flood_status` and `forecast_ts` - are used for assigning trend and prioritizing "duplicate" forecasts. These are now created in the RDS Bastion `postgresql_setup.sh.tftpl` script. * Two new database views - `rnr.stage_thresholds` and `rnr.flow_thresholds` - reorganize data in the external/foreign WRDS Ingest DB threshold table (external.threshold) for more efficient use here and eventually in other places (e.g. RnR). The SQL for these views was committed for the record in the `Core/LAMBDA/rnr_functions/rnr_domain_generator/sql/dba_stuff.sql` file, but I believe this view should get copied over with the dump of the RnR schema on deployments and thus should not need to be recreated manually. * Forecasts that are distributed as flow-only (i.e. have no associated rating curve to produce stage) are now also included as a value-added win (addressing issue #312). As a result of this: * The DB table produced by `rfc_max_forecast.sql` has new/modified column names, replacing every occurrence of "stage" with "value" (i.e. `max_stage_timestamp` to `max_value_timestamp`). * The `rfc_max_forecast.mapx` file was thus also modified to replace every occurrence of "stage" with "value" in both the "fieldName" and "alias" fields (i.e. `max_stage_timestamp` to `max_value_timestamp` and "Forecast Min Stage Timestamp" to "Forecast Min Value Timestamp") This work will also allow the Replace and Route to be somewhat redesigned to be completely in-sync with this new rfc_max_forecast service - likely using the underlying `publish.rfc_max_forecast` table as its starting point.
This PR, if merged, will replace the existing "rfc_max_forecast" service with a new version of the service whose backend data is now produced by direct access to the WRDS RFC Forecast database as opposed to the WRDS API. As a result: * The `every_five_minute` EventBridge rule that triggered the `viz-wrds-api-handler` Lambda has been modified to instead trigger the `initialize-viz-pipeline` Lambda directly. The viz-wrds-api-handler code is now deprecated and can be eventually removed. * The `initialize-viz-pipeline` Lambda code had to be modified to allow the "rfc" pipeline to be kicked off without referencing any input files - as there are none. This allows it to act on more of a "cron" basis and less of a "file-driven-event" basis. * The new `products/rfc/rfc_max_forecast.sql` query relies upon a few new database structures: * Two new ENUM types - `flood_status` and `forecast_ts` - are used for assigning trend and prioritizing "duplicate" forecasts. These are now created in the RDS Bastion `postgresql_setup.sh.tftpl` script. * Two new database views - `rnr.stage_thresholds` and `rnr.flow_thresholds` - reorganize data in the external/foreign WRDS Ingest DB threshold table (external.threshold) for more efficient use here and eventually in other places (e.g. RnR). The SQL for these views was committed for the record in the `Core/LAMBDA/rnr_functions/rnr_domain_generator/sql/dba_stuff.sql` file, but I believe this view should get copied over with the dump of the RnR schema on deployments and thus should not need to be recreated manually. * Forecasts that are distributed as flow-only (i.e. have no associated rating curve to produce stage) are now also included as a value-added win (addressing issue #312). As a result of this: * The DB table produced by `rfc_max_forecast.sql` has new/modified column names, replacing every occurrence of "stage" with "value" (i.e. `max_stage_timestamp` to `max_value_timestamp`). * The `rfc_max_forecast.mapx` file was thus also modified to replace every occurrence of "stage" with "value" in both the "fieldName" and "alias" fields (i.e. `max_stage_timestamp` to `max_value_timestamp` and "Forecast Min Stage Timestamp" to "Forecast Min Value Timestamp") This work will also allow the Replace and Route to be somewhat redesigned to be completely in-sync with this new rfc_max_forecast service - likely using the underlying `publish.rfc_max_forecast` table as its starting point.
There are a number of AHPS gauges that only report flow, and these are currently available in the WRDS API. Please add these flow-only gauges to the rfc_max_stage service, and update the code to use flow flood thresholds (major, moderate, etc.) for these locations. Because the service will now contain both stage and flow gauges, it will need a name update and possibly some attribute updates as well.
The text was updated successfully, but these errors were encountered: