The get_nldi_basin() function (from nhdplusTools) not working

[motasem-suleiman]

I keep getting an error when I use the get_nldi_basin() function from the nhdplusTools library.

Request failed [500]. Retrying in 1.2 seconds...
Request failed [500]. Retrying in 3.8 seconds...
Warning message:
Cannot open "{"description":"Something bad happened. Contact us with Reference Number: 1255184806","type":"error"}"; Check connection parameters.

[dblodgett-usgs]

This issue has been reported over here. internetofwater/nldi-services#389 Keep an eye on that for resolution.

[rburghol]

Hey @dblodgett-usgs -- FWIW I get the same error when using plot_nhdplus().
I have a workaround (realized I was using plot_nhdplus() to get tribs, which was foolish as I should have used navigate_nldi()), but this got me to wondering, is there a workaround if one were to download the NHD data for local storage using nhdplusTools, then pointing to the local dataset? Just a curiosity related to the concepts of caching/network efficiency a la #366

[dblodgett-usgs]

@rburghol and @motasem-suleiman -- we should be back in business. We had a bit of a comedy of version incompatibilities between AWS, Postgres and PostGIS that were quite confounding. Great work by @ewojtylko to chaise it down.

[mhweber]

@dblodgett-usgs I'm still seeing 403 request failed error with basic example from Get Started page - this code:

   library(nhdplusTools)
   library(sf)
   start_point <- st_sfc(st_point(c(-89.362239, 43.090266)), crs = 4269)
   start_comid <- discover_nhdplus_id(start_point)

   flowline <- navigate_nldi(list(featureSource = "comid", 
                           featureID = start_comid), 
                      mode = "upstreamTributaries", 
                      distance_km = 1000)

    subset_file <- tempfile(fileext = ".gpkg")
    subset <- subset_nhdplus(comids = as.integer(flowline$UT$nhdplus_comid),
                     output_file = subset_file,
                     nhdplus_data = "download", 
                     flowline_only = FALSE,
                     return_data = TRUE, overwrite = TRUE)

Returns this error for me:

    All intersections performed in latitude/longitude.
    Reading NHDFlowline_Network
    Spherical geometry (s2) switched off
    Request failed [403]. Retrying in 1 seconds...
    Request failed [403]. Retrying in 3.1 seconds...
    Error in out[[1]] : subscript out of bounds
    Spherical geometry (s2) switched on

[dblodgett-usgs]

Yeah -- The firewall issue that kicked off all this was fixed in one way but the example you reference is still not quite working. We are working on opening up the firewall to larger POST payloads and then it should be fully back.

[rburghol]

Ahh, I noticed this was a throttling issue yesterday, 50 segments seemed to be acceptable to the service, so I made this workaround using get_nhdplus() @mhweber HTH:

flowline <- navigate_nldi(
  list(featureSource = "comid", featureID = outlet_comid), 
  mode = "UT", 
  distance_km = 1000
)
nhd <- FALSE
totes <- 0
ncat <- nrow(flowline$UT_flowlines)
i <- 0
while (totes < ncat) {
  brow <- i * 50 + 1
  erow <- min( (brow+49), ncat)
  nhd_batch <- get_nhdplus(comid = flowline$UT_flowlines$nhdplus_comid[brow:erow])
  if (is.logical(nhd)) {
    nhd <- nhd_batch
  } else {
    nhd <- rbind(nhd, nhd_batch)
  }
  totes <- totes + nrow(nhd_batch)
  i <- i + 1
}

[dblodgett-usgs]

Nice hack! That would totally work around what we are seeing. I'll update when we get this figured with the security folks.

[dblodgett-usgs]

An exception was put in place last night. Seems that we are back in business. Tests for me work. Will leave this open till I get confirmation you all are back in action. Thanks for your patience.

[mhweber]

All working when I check- thanks @dblodgett-usgs !