Scaling CONUS404 spatial aggregation to geofabric catchements over all of CONUS

[amsnyder]

@ted80810 will be spatially aggregating the CONUS404 data to geofabric stream segments for all of CONUS. He plans to use this notebook as a starting point, but I anticipate there may be challenges when scaling this computation up. I'm opening this issue to brainstorm approaches to parallelizing this workflow and discuss any challenges that come up.

[sfoks]

Whoa this is massive computational problem, like aggregating like all state and flux variables upstream and intersecting a segment? Are there catchments already delineated to geospatial fabric segments? Andy Bock or Dave B would know.

[rmcd-mscb]

What does aggregating to a stream segment mean? Spatial aggregating to the stream-segment catchment or interpolating the data to the stream-segment itself?

[amsnyder]

I don't know which variables we need yet, but maybe @ted80810 does!

When I say "aggregation to a stream segment" I mean area-weighted aggregation to a catchment that can be matched to a stream segment.

We will use catchments that have already been delineated, but I think we are still checking on the exact set of features to use. I believe it will be the geofabric catchments used by NHM - but I am no expert on this dataset, so I couldn't point you to the feature set. I think @ted80810 or maybe @lekoenig were checking on the features.

[rmcd-mscb]

We've mapped c404 and c404bc to gfv1.1. I don't think this should be a big computing issue. You could use gdptools - just saying :) Would be happy to help Ted set that up.

[rmcd-mscb]

Here is an example notebook run on tallgrass mapping c404bc to Upper Colorado fabric: https://nbviewer.org/gist/rmcd-mscb/12f0e01dd2dcd8aa9f8a85444d530476

I can provide an environment file.

[ted80810]

@rmcd-mscb It would be great to utilize gdp tools, so I will ping you very soon.

I don't have a definitive list of the variables as yet, but I will certainly share it as soon as it is confirmed.

[rmcd-mscb]

Sounds good @ted80810 I'll be out tomorrow but around all next week - Would be happy to help out!

[amsnyder]

@rmcd-mscb when you say've mapped conus404 to gfv1.1, so you mean you've generated the weights matrix already for all of CONUS?

[rmcd-mscb]

I've generated the weights for gfv1.1 and I've mapped 40 years of data. It's being used with NHM currently.

[amsnyder]

That's great! I'd be curious to hear how long it took to run the aggregation for the Upper Colorado basin. And also just want to confirm that NHM doesn't already have an aggregated version of the data somewhere (for all of CONUS) - and that we should in fact start with the weight matrix.

Also where is the weight matrix being stored currently?

[rmcd-mscb]

The gfv1.1 c404bc data that are used with NHM are here: /caldera/projects/usgs/water/wbeep/onhm_dev/climate/c404bc

Those files are formatted for NHM but they should be generally useful to gfv1.1 applications. Also, I think it took somewhere on the order of 4-6 hours to do the aggregation for CONUS. It takes several minutes to generate the weights.

[amsnyder]

Just making a note that @ted80810 is working on this for another project, but I would love to have his work contributed back to this repo once it is finished. I'll keep this issue open and I've assigned it to @ted80810 so we don't lose track of this work.