Google Floods Part 1: Adding backend + creating Gauge Points; COUNTRY=cambodia #1328

lowriech · 2024-08-08T22:11:17Z

Description

This fulfills step 1 of 3 for #1317 (see plan).

In this PR, we're fetching flood status by gauge from Google's Floods API for the entire country (10-100 gauges per country). We're displaying these gauges as points on Prism. They're color coded by flood status and provide additional insight when clicked.

This PR also adds support to provide a popup title in the layer config. Because these gauges are typically located on rivers that server as the boundary between two regions, using the region title for the pop up doesn't make much sense. Instead, we're adding an approach that allows one to use formatted feature properties as the title.

Forecasts for each gauge are added here: #1330

How to test the feature:

Open the Mozambique deployment
Enable the Google Flood layer

Checklist - did you ...

Test your changes with

REACT_APP_COUNTRY=rbd yarn start
REACT_APP_COUNTRY=cambodia yarn start
REACT_APP_COUNTRY=mozambique yarn start
Add / update necessary tests?
Add / update outdated documentation?

Screenshot/video of feature:

github-actions · 2024-08-08T22:18:28Z

Build succeeded and deployed at https://prism-1328.surge.sh
(hash 6d33543 deployed at 2024-10-31T19:46:19)

ericboucher · 2024-08-09T20:30:39Z

api/app/main.py

@@ -412,3 +413,10 @@ def post_raster_geotiff(raster_geotiff: RasterGeotiffModel):
    return JSONResponse(
        content={"download_url": presigned_download_url}, status_code=200
    )
+
+
+@app.get("/google-floods-gauges")


If we're gonna add a bunch of integrations, it will be worthwile to think about a standardized approach and common architecture. And maybe expose them through a sub endpoint of the API to keep things cleaner. wdyt @lowriech ?

I think that's a good question, @ericboucher. Do you see other APIs we're already leveraging fitting into this integration category (kobo, hdc stats, others)? Or are you thinking just for new integrations?

I'm wary of premature abstraction and trying to determine if I think now is the right time to align on this standardized approach.

It seems like low-hanging fruit to abstract once we decide on a pattern. So yes, agree we should think about it, but not sure about the velocity of adding entirely new providers. Would vote to wait until we see more pattern around this.

gislawill · 2024-08-12T05:28:39Z

frontend/src/config/mozambique/layers.json

+    "title": "Google Flood Gauges",
+    "type": "point_data",
+    "hex_display": false,
+    "data": "http://0.0.0.0:80/google-floods/gauges/?region_code=MZ",


Note: need a better development set up

Set to a relative path, make sure we're using DEFAULT_API_URL for these requests

@ericboucher have you all deployed the prism-api out in surge-like deployments before for testing?

gislawill · 2024-08-12T22:10:28Z

@lowriech and @ericboucher, there are a couple outstanding issues (noted in the PR description) but this ready for initial review

gislawill · 2024-08-12T22:20:23Z

frontend/src/components/MapView/utils.ts

+      if (
+        itemProps.visibility === FeatureInfoVisibility.IfDefined &&
+        !properties[item]
+      ) {
+        return obj;
+      }


We have some feature properties that may or may not be defined by the google api (for example: site name, river name). This allows us to hide the property from the pop up if it's not defined

gislawill · 2024-08-12T22:22:53Z

api/app/googleflood.py

+            "river": (
+                data["river"] if "river" in data and len(data["river"]) > 1 else None
+            ),


"river" can come in as a string with just 1 space (" "). This ensures there's a real value

ericboucher · 2024-08-13T09:36:18Z

frontend/src/config/mozambique/layers.json

+      },
+      "gaugeId": {
+        "type": "text",
+        "template": "Gauge ID: %s",


Let's think of a way to make this template translatable?

How have we done this in other parts of the app? Can we use a translate wrapper function within the tooltips?

We haven't really used templates in PRISM yet. We usually consider the template as a title and then handle the title and the data separately. So instead of template you would just have "title": "Gauge ID". and then append the value. You can have a look at the PopupContent file for examples. If we think the structure will be simple like that, they that's probably ok.

Another option is to use the translation mapping and have the template be: Gauge ID: {id}, then use it like this:

<Trans values={{ id }}> <Typography>{t(template)}</Typography> </Trans>

Thanks @ericboucher! I'm leaning towards the latter option (translation mapping). We used these templates in react-i18next pretty extensively in the distant past (my Hilton Hotels days). Once we started using them, it gave us a ton of flexibility in terms of copy. If you're ok with it, I'll implement it this way.

gislawill · 2024-08-16T15:39:28Z

Note: I'm adding pytest-recording to store our google flood responses in local tests and use them in CI/CD. That will fix the failing API test in this PR

…api key to docker compose

…AM/prism-app into feature/google-floods/1317

api/app/googleflood.py

api/app/tests/cassettes/test_google_floods_api/test_get_google_floods_gauges_api.yaml

wadhwamatic · 2024-10-07T20:34:56Z

Great to see this nearly complete - thanks @gislawill. I mentioned the map interaction bug that's preventing a full test but hopefully that is resolved soon.

In the meantime, in addition to displaying the gauge ID, can we first show the river name? I see that in Flood Hub interface, hopefully it's also exposed via the API.

Note, we'd need to translate that text as well. I see that Flood Hub offers several language options including Khmer. Ex: https://sites.research.google/floods/e/l/10.96043262736797/103.63535499532549/8.45328318786621/g/hybas_4121148610?layers&hl=km

hl=km

gislawill · 2024-10-07T20:57:42Z

In the meantime, in addition to displaying the gauge ID, can we first show the river name? I see that in Flood Hub interface, hopefully it's also exposed via the API.
Note, we'd need to translate that text as well. I see that Flood Hub offers several language options including Khmer. Ex: https://sites.research.google/floods/e/l/10.96043262736797/103.63535499532549/8.45328318786621/g/hybas_4121148610?layers&hl=km

Hey @wadhwamatic, the river name is sometimes provided by the API but often it is not. I've currently set this up so we display the site name if it's provided. If not, we display the river name if it's provided. If not, we display the gauge id (which is always provided). How does that logic sound to you?

Here's how it looks:

This PR	Next PR

I tried to find the translated names in the API but unfortunately there's no option in the API documentation and the API fails if the hl parameter is included in the request.

I'll update this title component so that at least "site name:" and "river:" can be translated even in the API provided name won't be.

wadhwamatic · 2024-10-07T22:05:22Z

Hey @wadhwamatic, the river name is sometimes provided by the API but often it is not. I've currently set this up so we display the site name if it's provided. If not, we display the river name if it's provided. If not, we display the gauge id (which is always provided). How does that logic sound to you?

That logic sounds good - thanks! Is there an example of site name in Cambodia?

That's too bad about the translation via the API but thanks for noting it regardless.

gislawill · 2024-10-09T21:33:06Z

That logic sounds good - thanks! Is there an example of site name in Cambodia?

There is not unfortunately. A bunch of rivers but no site names

gislawill · 2024-10-10T03:02:40Z

Heads up @wadhwamatic, the today "Date" is now set on these layers

I'll work with @ericboucher to get a new server deployment out for testing

gislawill · 2024-10-10T14:49:29Z

Server deployment containing the updates is out (thanks @ericboucher for walking me through it). @wadhwamatic you can resume testing

wadhwamatic · 2024-10-10T20:19:44Z

@gislawill - thanks for adding the time dimension here. I tested date intersections a few ways and this is looking good from my side, as does the river gauge info noted in this PR

gislawill · 2024-10-24T20:04:38Z

Remaining todo items:

Ensure all parts of tooltip are translate-able

* Adding graph for google flood data * Merge conflict resolution and units * Updating tests * formatting * Using superscript * Supporting rbd * Add support for Cambodia * Updating url for data * api lint * Fixing date logic on graph look up * Updates to chart to denote past vs future * Adding translation to 'future' label * Fix chart tests using mock * Requested changes: descriptive labels

Adding Google Floods Backend

c5c1215

lowriech marked this pull request as draft August 8, 2024 22:11

ericboucher reviewed Aug 9, 2024

View reviewed changes

gislawill added 2 commits August 11, 2024 22:26

Viewing the gauges on the frontend, adding error handling to backend

b9722d3

Update comment

bc92d38

gislawill reviewed Aug 12, 2024

View reviewed changes

Updates to support tooltip content

08a4499

gislawill changed the title ~~Adding Google Floods Backend~~ Adding Google Floods Backend + Creating a Gauge Points Aug 12, 2024

gislawill added 3 commits August 12, 2024 14:05

copy update

0653f1e

Refactor title state management

7cd866d

Fixing and adding server tests

94e1588

gislawill changed the title ~~Adding Google Floods Backend + Creating a Gauge Points~~ Adding Google Floods Backend + Creating a Gauge Points; COUNTRY=mozambique Aug 12, 2024

gislawill reviewed Aug 12, 2024

View reviewed changes

ericboucher reviewed Aug 13, 2024

View reviewed changes

ericboucher and others added 4 commits August 13, 2024 14:19

update set_env

6ac3aad

Merge branch 'master' into feature/google-floods/1317

eb29b6e

update snapshots

6be26d7

Merge branch 'master' into feature/google-floods/1317

2fe481a

gislawill linked an issue Aug 19, 2024 that may be closed by this pull request

[Feature Request]: Google Flood Hub integration #1317

Open

gislawill added 3 commits August 20, 2024 17:29

Adding support for multi country deployments and adding google flood …

8e84cbb

…api key to docker compose

Use interpolation for popup titles

efdf2c9

Adding pytest-recording

8fa327c

gislawill marked this pull request as ready for review August 21, 2024 01:21

ericboucher added 3 commits August 21, 2024 10:38

Merge branch 'master' into feature/google-floods/1317

f5ffbf6

Update test_google_floods_api.py

438ee04

Merge branch 'feature/google-floods/1317' of https://github.com/WFP-V…

e369704

…AM/prism-app into feature/google-floods/1317

gislawill changed the title ~~Adding Google Floods Backend + Creating a Gauge Points; COUNTRY=mozambique~~ Adding Google Floods Backend + Creating Gauge Points; COUNTRY=mozambique Aug 21, 2024

gislawill reviewed Aug 21, 2024

View reviewed changes

api/app/googleflood.py Outdated Show resolved Hide resolved

gislawill reviewed Aug 21, 2024

View reviewed changes

api/app/tests/cassettes/test_google_floods_api/test_get_google_floods_gauges_api.yaml Outdated Show resolved Hide resolved

gislawill and others added 4 commits August 21, 2024 21:31

Fixing river template

6547f3b

More selective with pytest recording

ac58dca

Merge branch 'master' into feature/google-floods/1317

65af7e7

Update API URL

0adb49d

gislawill requested a review from wadhwamatic August 30, 2024 22:05

gislawill mentioned this pull request Sep 24, 2024

[Tech Debt]: Create cassettes for external API calls in pytest #1348

Open

ericboucher and others added 2 commits September 25, 2024 11:26

Merge branch 'master' into feature/google-floods/1317

dc94f60

Merge branch 'master' into feature/google-floods/1317

b0780fa

gislawill changed the title ~~Adding Google Floods Backend + Creating Gauge Points; COUNTRY=mozambique~~ Adding Google Floods Backend + Creating Gauge Points; COUNTRY=cambodia Oct 7, 2024

Adding support for Cambodia

7f401bb

gislawill added 3 commits October 7, 2024 15:07

Merge branch 'master' into feature/google-floods/1317

341a3a7

Merge branch 'master' into feature/google-floods/1317

1e51479

Adding date support

2a890cf

gislawill changed the title ~~Adding Google Floods Backend + Creating Gauge Points; COUNTRY=cambodia~~ Google Floods Part 1: Adding backend + creating Gauge Points; COUNTRY=cambodia Oct 24, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Google Floods Part 1: Adding backend + creating Gauge Points; COUNTRY=cambodia #1328

Google Floods Part 1: Adding backend + creating Gauge Points; COUNTRY=cambodia #1328

lowriech commented Aug 8, 2024 •

edited by gislawill

Loading

github-actions bot commented Aug 8, 2024 •

edited

Loading

ericboucher Aug 9, 2024

gislawill Aug 12, 2024

lowriech Aug 13, 2024

gislawill Aug 12, 2024 •

edited

Loading

gislawill Aug 16, 2024

gislawill commented Aug 12, 2024

gislawill Aug 12, 2024

gislawill Aug 12, 2024

ericboucher Aug 13, 2024

lowriech Aug 13, 2024

ericboucher Aug 15, 2024 •

edited

Loading

gislawill Aug 16, 2024

gislawill Aug 21, 2024

gislawill commented Aug 16, 2024

wadhwamatic commented Oct 7, 2024

gislawill commented Oct 7, 2024

wadhwamatic commented Oct 7, 2024

gislawill commented Oct 9, 2024

gislawill commented Oct 10, 2024

gislawill commented Oct 10, 2024

wadhwamatic commented Oct 10, 2024

gislawill commented Oct 24, 2024

Google Floods Part 1: Adding backend + creating Gauge Points; COUNTRY=cambodia #1328

Are you sure you want to change the base?

Google Floods Part 1: Adding backend + creating Gauge Points; COUNTRY=cambodia #1328

Conversation

lowriech commented Aug 8, 2024 • edited by gislawill Loading

Description

How to test the feature:

Checklist - did you ...

Screenshot/video of feature:

github-actions bot commented Aug 8, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gislawill Aug 12, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gislawill commented Aug 12, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ericboucher Aug 15, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gislawill commented Aug 16, 2024

wadhwamatic commented Oct 7, 2024

gislawill commented Oct 7, 2024

wadhwamatic commented Oct 7, 2024

gislawill commented Oct 9, 2024

gislawill commented Oct 10, 2024

gislawill commented Oct 10, 2024

wadhwamatic commented Oct 10, 2024

gislawill commented Oct 24, 2024

lowriech commented Aug 8, 2024 •

edited by gislawill

Loading

github-actions bot commented Aug 8, 2024 •

edited

Loading

gislawill Aug 12, 2024 •

edited

Loading

ericboucher Aug 15, 2024 •

edited

Loading