Issue with RCS layer registration

[mweech]

@yblain commented on Tue Jun 20 2017

A bug has been identified with regards to the registration of a service in RCS.

Here is the description of the problem from Jose's point of view:

Hi

I have been reported a problem with registering the following layer in RCS:

http://services1.arcgis.com/HsjBaDykC1mjhXz9/ArcGIS/rest/services/FloodExtentPolygon_SouthManitoba_MODIS_20170405_1530_1705/FeatureServer/0

The Data Catalogue is sending the following request:

{
"en": {
"service_url": "http://services1.arcgis.com/HsjBaDykC1mjhXz9/ArcGIS/rest/services/FloodExtentPolygon_SouthManitoba_MODIS_20170405_1530_1705/FeatureServer/0",
"service_type": "esriFeature",
"service_name": "FloodExtentPolygon_SouthManitoba_MODIS_20170405_1530_1705"
},
"fr": {
"service_url": "http://services1.arcgis.com/HsjBaDykC1mjhXz9/ArcGIS/rest/services/FloodExtentPolygon_SouthManitoba_MODIS_20170405_1530_1705/FeatureServer/0",
"service_type": "esriFeature",
"service_name": "FloodExtentPolygon_SouthManitoba_MODIS_20170405_1530_1705"
},
"version": "2.0"
}

But getting the following error:

{ "msg": "Problem communicating with service endpoint http://services1.arcgis.com/HsjBaDykC1mjhXz9/ArcGIS/rest/services/FloodExtentPolygon_SouthManitoba_MODIS_20170405_1530_1705/FeatureServer/0 content-type" }

Tested also with the RCS test page and same issue.

The request looks fine, but maybe not sure if maybe the service_type value is not what is expecting RCS?

If you can provide me some feedback on how to manage this is really appreciated, thanks.

The related issue in Redmine is this one: http://redmine.gcgeo.gc.ca/redmine/issues/9156

Regards,
Jose García

[mweech]

moved to RCS repo.

[yblain]

We are getting the same issue with another layer from one of the partners. The commonality seems to be that both of these layers are coming from AGOL.

Here i s the payload that we are trying to register:

{
"version": "2.0",
"fr": {
"service_url": "http://services2.arcgis.com/wCOMu5IS7YdSyPNx/arcgis/rest/services/campsites_sites_camping_vw/FeatureServer/0",
"service_type": "esriFeature",
"service_name": "Sites de camping",
"metadata": {
"catalogue_url": "https://gcgeo.gc.ca/geonetwork/metadata/fre/74054d44-68cf-41af-8919-5f09f80dcd02",
"metadata_url": "https://gcgeo.gc.ca/geonetwork/srv/fre/xml.metadata.download?uuid=74054d44-68cf-41af-8919-5f09f80dcd02&fromWorkspace="
}
},
"en": {
"service_url": "http://services2.arcgis.com/wCOMu5IS7YdSyPNx/arcgis/rest/services/campsites_sites_camping_vw/FeatureServer/0",
"service_type": "esriFeature",
"service_name": "Campsites",
"metadata": {
"catalogue_url": "https://gcgeo.gc.ca/geonetwork/metadata/eng/74054d44-68cf-41af-8919-5f09f80dcd02",
"metadata_url": "https://gcgeo.gc.ca/geonetwork/srv/eng/xml.metadata.download?uuid=74054d44-68cf-41af-8919-5f09f80dcd02&fromWorkspace="
}
}
}

Here is the log info we are getting back:

2017-12-05T16:27:03.948Z {"msg": "Problem communicating with service endpoint http://services2.arcgis.com/wCOMu5IS7YdSyPNx/arcgis/rest/services/campsites_sites_camping_vw/FeatureServer/0 content-type"}

2017-12-05T16:27:03.948Z put failed 400

2017-12-05T16:27:03.078Z HMAC_SHA256(msg,key): PKoX3MNIlor0okpzRfM4lnoAwj4AoS9mZ6eaL84N5qs

2017-12-05T16:27:03.072Z key: test_-k

2017-12-05T16:27:03.071Z msg: /v2/register/74054d44-68cf-41af-8919-5f09f80dcd02jstest2017-12-05T16:27:03.070Z{ "version": "2.0", "fr": { "service_url": "http://services2.arcgis.com/wCOMu5IS7YdSyPNx/arcgis/rest/services/campsites_sites_camping_vw/FeatureServer/0", "service_type": "esriFeature", "service_name": "Sites de camping", "metadata": { "catalogue_url": "https://gcgeo.gc.ca/geonetwork/metadata/fre/74054d44-68cf-41af-8919-5f09f80dcd02", "metadata_url": "https://gcgeo.gc.ca/geonetwork/srv/fre/xml.metadata.download?uuid=74054d44-68cf-41af-8919-5f09f80dcd02&fromWorkspace=" } }, "en": { "service_url": "http://services2.arcgis.com/wCOMu5IS7YdSyPNx/arcgis/rest/services/campsites_sites_camping_vw/FeatureServer/0", "service_type": "esriFeature", "service_name": "Campsites", "metadata": { "catalogue_url": "https://gcgeo.gc.ca/geonetwork/metadata/eng/74054d44-68cf-41af-8919-5f09f80dcd02", "metadata_url": "https://gcgeo.gc.ca/geonetwork/srv/eng/xml.metadata.download?uuid=74054d44-68cf-41af-8919-5f09f80dcd02&fromWorkspace=" } } }

2017-12-05T16:12:10.601Z {"readyState":4,"responseText":"null","responseJSON":null,"status":404,"statusText":"NOT FOUND"}

2017-12-05T16:12:10.601Z get failed 404

2017-12-05T16:12:10.320Z GET URL: /v2/doc/en/74054d44-68cf-41af-8919-5f09f80dcd02

[barryytm]

After some investigations using the JSON that @yblain provided, I've discovered a few things

1. line 57 in universal.py was trying to access content-type in r.header which caused an error because it wasn't defined
2. The meta data was also causing an error [Errno 1] _ssl.c:510: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed. It still requires more investigations, however it's not the main problem so will investigate it later
3. Although the JSON @yblain provided stated it was V2 (version 2). The code in RCS was actually creating a V1 (version 1). This seems to be the main reason why the layer failed.

Here are some inputs from @james-rae about point number 3

• Existing code is broken for FeatureServer, as it depends on /legend endpoint

• The code using this is generating a symbology node on a final product that gets stored in RCS. This code is in make_v1_feature_node, which supports RAMP v1. You'll notice RAMP v2 does not use symbology in it's layer config

Initial Idea (needs feedback from Aly / Dan): I'm not quite sure what make_v1_feature_node is used for big-picture. I'm assuming it's if somebody does a GET in RCS for a v1 style entry, then it returns this object instead of the v2 one. I think we should consider skipping the symbology part for make_v1_feature_node if it's a FeatureServer (or just fails in general). The result, if I have it correct, would mean a v2 entry will be ok for RAMP v2, but if a version of RAMP v1 tries to consume that RCS key, it will be missing it's symbology. This will look ugly, but RAMP v1 was never promised to work for FeatureServer anyways.

After reading what @james-rae and did some testings. I have found that replacing this with esri.make_feature_node(json_request[lang]) would work. Maybe @alyec would have some inputs on it? 📦

[barryytm]

The error with creating a v1 node was caused by requesting the legend of the layer from the server which it did not exit (link). It seems commenting out node['symbology'] would also work.

There are 2 solutions I can think of as for now

• Use make_feature_node instead of make_v1_feature_node (i.e. make a v2 node instead of a v1 node)
• Check whether the service is coming from a Map Server or a Feature Server. If it is coming from Feature Server, then RCS would avoid running node['symbology']

Personally I think the second option is better, simply because we will still be able get a v1 style entry using GET.

[james-rae]

A third option, which might prevent a number of errors in RAMP v1, is if a Feature Server is incoming, the RCS v1 code inserts a placeholder in the symbology node (e.g. A question-mark graphic and the word "Unknown" for text).

But before doing this, I think we need to decide if RAMP v1 requesting an RCS key that refers to a Feature Server is a valid use case. If not, is easier just to skip the generation of RCS v1 entries for Feature Server (or something similar).

[mweech]

@james-rae you have a good point re: we probably would never run into v1 entries for Feature Server being requested unless we get an odd configuration mismatch of DC and RCS/RAMP.

[barryytm]

There are 3 issues related to Metadata/Catalog that I have identified using the JSON @yblain provided

1. The certificates failed when requesting from metadata/catalog URLs. However I was able to bypassed this issue with setting verify=False in link but it should not be a permenant solution as it exposes vulnerability. Is this issue due to running from Vagrant (was running from Vagrant) or could it have something to do with the security setting on the server?

   • Error message from Metadata:
     { "msg": "Metadata URL: \"https://gcgeo.gc.ca/geonetwork/srv/eng/xml.metadata.download?uuid=74054d44-68cf-41af-8919-5f09f80dcd02&fromWorkspace=\" request failed with error \"[Errno 1] _ssl.c:510: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed\"" }
   • Error message from Catalog:
     { "msg": "Catalogue URL: \"https://gcgeo.gc.ca/geonetwork/metadata/eng/74054d44-68cf-41af-8919-5f09f80dcd02\" request failed with error \"[Errno 1] _ssl.c:510: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed\"" }

2. The MIME type of requested data from the provided metadata enpoint was not what RCS expected it would be. RCS was expecting the MIME type to be xml; what it got back was octet-stream. Should we be supporting octet-stream, charset=UTF-8?

   • Error message from Metadata: { "msg": "Metadata URL: \"https://gcgeo.gc.ca/geonetwork/srv/eng/xml.metadata.download?uuid=74054d44-68cf-41af-8919-5f09f80dcd02&fromWorkspace=\" could not be retrieved: expected \"['application/xml', 'text/xml']\", got \"application/octet-stream;charset=UTF-8\"" }

3. The request from the provided Catalog URL failed. A couple things to point out. The link returned a valid page when pasted in the browser, and the MIME type what RCS expected was html from a Catalog endpoint.

   • Error message from Catalog: { "msg": "Catalogue URL: \"https://gcgeo.gc.ca/geonetwork/metadata/eng/74054d44-68cf-41af-8919-5f09f80dcd02\" could not be retrieved: bad HTTP status code received 404" }

[dan-bowerman]

1. I've experienced this in production as well (not in a Vagrant environment), and have no idea what the solution could be.
2. We've experience this MIME type issue before -- generally speaking it's the server being misconfigured that is the problem. Since this keeps cropping up, I'm not sure if RCS should just be more lenient with filetypes it consumes? @mweech, @alyec?
3. This is coming from your local Vagrant RCS? The URL works fine for me too, locally, so it's unusual that it would return a 404

[alyec]

1. Probably a stale CA store, CA certs probably need to be upgraded. Destroying and recreating the vagrant env should help in Barry's case, if not maybe some change to the Vagrant config is in order. For production we can sort out in a separate issue, but my first question will be if the server is fully patched.

2. Rather than expand the list to include octet-stream I'd rather put in a config option to drop the check if we know a server isn't going to give back useful data via MIME types. octet-stream is far too general to draw conclusions from IMO and I don't think it really gives us any additional confidence that the viewer isn't going to have issues accessing the service.

3. I spent a bit more time on this than I probably should have. There's some really odd behaviour and it seems the Accept header is the culprit.

This fails:

curl -v --http1.1 -i "https://gcgeo.gc.ca/geonetwork/metadata/eng/74054d44-68cf-41af-8919-5f09f80dcd02"

but this works:

curl -v --http1.1 -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8" -i "https://gcgeo.gc.ca/geonetwork/met adata/eng/74054d44-68cf-41af-8919-5f09f80dcd02"

In the first case the header goes out as Accept: */* which I can't see being objectionable to the server, but I guess it is. I think we should try to figure out a bit more of what's going on within the server.

[jvanulde]

So is this closed?

[jvanulde]

Upgraded to v2.3.0 and still can't register:

http://services2.arcgis.com/wCOMu5IS7YdSyPNx/arcgis/rest/services/campsites_sites_camping_vw/FeatureServer/0

[mweech]

This service adds fine on our test RCS instances:

{"fr":{"service_url":"http://services1.arcgis.com/HsjBaDykC1mjhXz9/ArcGIS/rest/services/FloodExtentPolygon_SouthManitoba_MODIS_20170405_1530_1705/FeatureServer/0","service_type":"esriFeature","service_name":"Flood Extent FR"},"en":{"service_url":"http://services1.arcgis.com/HsjBaDykC1mjhXz9/ArcGIS/rest/services/FloodExtentPolygon_SouthManitoba_MODIS_20170405_1530_1705/FeatureServer/0","service_type":"esriFeature","service_name":"Flood Extent"},"version":"2.0"}

http://fgpv.cloudapp.net/demo/develop/prod/samples/index-fgp-en.html?keys=floodextent

[barryytm]

I did't have any troubles registering the camp sites layer.

Here is the snippet:

{  
   "version":"2.0",
   "fr":{  
      "service_url":"http://services2.arcgis.com/wCOMu5IS7YdSyPNx/arcgis/rest/services/campsites_sites_camping_vw/FeatureServer/0",
      "service_type":"esriFeature",
      "service_name":"Sites de camping"
   },
   "en":{  
      "service_url":"http://services2.arcgis.com/wCOMu5IS7YdSyPNx/arcgis/rest/services/campsites_sites_camping_vw/FeatureServer/0",
      "service_type":"esriFeature",
      "service_name":"Campsites"
   }
}

@jvanulde What did the error message say?

[jvanulde]

2018-08-24T19:16:53.753Z {"msg": "Problem communicating with service endpoint http://services2.arcgis.com/wCOMu5IS7YdSyPNx/arcgis/rest/services/campsites_sites_camping_vw/FeatureServer/0 No JSON object could be decoded"}

2018-08-24T19:16:53.753Z put failed 400

[jvanulde]

Still an issue. See log: http://redmine.gcgeo.gc.ca/redmine/attachments/download/5481/rcs.log where this error is provided:

 FeatureServer registration must specify a feature layer