Trouble getting going on Windows

[garethsb]

I have switched from the RiedelCommunications repo, successfully pip installed the additional dependencies (gitpython, netifaces, ramlfication), and started up test-nmos.py OK. I can see it's fetched the IS-04 specs etc.

Trying to execute first test, "IS-04 Node API", I get error in the browser and console:

jsonref.JsonRefError: URLError: <urlopen error [WinError 2] The system cannot find the file specified: '\\resource_core.json'>

Since it has successfully downloaded resource_core.json and the other schema files, I assume this is about resolving the first '$ref' it processes, presumably in schemas/node.json?

Any ideas?

[andrewbonney]

Hi Gareth,
Do you have the full traceback for this error? I'd just like to try and narrow down where exactly in the code it's happening. It looks like a simple Linux/Windows file path error, but it's not clear from that whether it's happening in something I have control of, or in an external library.

[garethsb]

Traceback (most recent call last):
  File "...\AppData\Local\Programs\Python\Python37\lib\site-packages\flask\app.py", line 2309, in __call__
    return self.wsgi_app(environ, start_response)
  File "...\AppData\Local\Programs\Python\Python37\lib\site-packages\flask\app.py", line 2295, in wsgi_app
    response = self.handle_exception(e)
  File "...\AppData\Local\Programs\Python\Python37\lib\site-packages\flask\app.py", line 1741, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "...\AppData\Local\Programs\Python\Python37\lib\site-packages\flask\_compat.py", line 35, in reraise
    raise value
  File "...\AppData\Local\Programs\Python\Python37\lib\site-packages\flask\app.py", line 2292, in wsgi_app
    response = self.full_dispatch_request()
  File "...\AppData\Local\Programs\Python\Python37\lib\site-packages\flask\app.py", line 1815, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "...\AppData\Local\Programs\Python\Python37\lib\site-packages\flask\app.py", line 1718, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "...\AppData\Local\Programs\Python\Python37\lib\site-packages\flask\_compat.py", line 35, in reraise
    raise value
  File "...\AppData\Local\Programs\Python\Python37\lib\site-packages\flask\app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "...\AppData\Local\Programs\Python\Python37\lib\site-packages\flask\app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "...\nmos-testing\nmos-test.py", line 98, in index_page
    test_obj = IS0401Test.IS0401Test(base_url, apis, spec_versions, version, spec_path, REGISTRY)
  File "...\nmos-testing\IS0401Test.py", line 37, in __init__
    GenericTest.__init__(self, base_url, apis, spec_versions, test_version, spec_path)
  File "...\nmos-testing\GenericTest.py", line 50, in __init__
    self.parse_RAML()
  File "...\nmos-testing\GenericTest.py", line 60, in parse_RAML
    self.apis[api]["spec"] = Specification(os.path.join(self.spec_path + '/APIs/' + self.apis[api]["raml"]))
  File "...\nmos-testing\Specification.py", line 44, in __init__
    resource_data["responses"][response.code] = self._extract_response_schema(response, file_path)
  File "...\nmos-testing\Specification.py", line 114, in _extract_response_schema
    return self._deref_schema(os.path.dirname(file_path), schema=self.global_schemas[schema_loc])
  File "...\nmos-testing\Specification.py", line 138, in _deref_schema
    return process(schema)
  File "...\nmos-testing\Specification.py", line 125, in process
    return {k: process(v) for k, v in obj.items()}
  File "...\nmos-testing\Specification.py", line 125, in <dictcomp>
    return {k: process(v) for k, v in obj.items()}
  File "...\nmos-testing\Specification.py", line 127, in process
    return [process(x) for x in obj]
  File "...\nmos-testing\Specification.py", line 127, in <listcomp>
    return [process(x) for x in obj]
  File "...\nmos-testing\Specification.py", line 122, in process
    if isinstance(obj, dict):
  File "...\AppData\Local\Programs\Python\Python37\lib\site-packages\proxytypes.py", line 175, in __getattribute__
    return getattr(self.__subject__, attr)
  File "...\AppData\Local\Programs\Python\Python37\lib\site-packages\proxytypes.py", line 176, in __getattribute__
    return _oga(self, attr)
  File "...\AppData\Local\Programs\Python\Python37\lib\site-packages\proxytypes.py", line 134, in wrapper
    return method(self, *args, **kwargs)
  File "...\AppData\Local\Programs\Python\Python37\lib\site-packages\proxytypes.py", line 254, in __subject__
    self.cache = super(LazyProxy, self).__subject__
  File "...\AppData\Local\Programs\Python\Python37\lib\site-packages\proxytypes.py", line 134, in wrapper
    return method(self, *args, **kwargs)
  File "...\AppData\Local\Programs\Python\Python37\lib\site-packages\proxytypes.py", line 240, in __subject__
    return self.callback()
  File "...\AppData\Local\Programs\Python\Python37\lib\site-packages\proxytypes.py", line 134, in wrapper
    return method(self, *args, **kwargs)
  File "...\AppData\Local\Programs\Python\Python37\lib\site-packages\jsonref.py", line 180, in callback
    self._error("%s: %s" % (e.__class__.__name__, unicode(e)), cause=e)
  File "...\AppData\Local\Programs\Python\Python37\lib\site-packages\proxytypes.py", line 134, in wrapper
    return method(self, *args, **kwargs)
  File "...\AppData\Local\Programs\Python\Python37\lib\site-packages\jsonref.py", line 229, in _error
    cause=cause,
jsonref.JsonRefError: URLError: <urlopen error [WinError 2] The system cannot find the file specified: '\\resource_core.json'>

[andrewbonney]

Could you try adding the following to line 120 of Specification.py (just inside def _deref_schema(self, dir, name=None, schema=None):)? If that sorts it out I'll work on a better fix...

file_prefix = "file:///" if os.name == "nt" else "file:"
if not dir.startswith("file:"):
    dir = file_prefix + dir

[garethsb]

Thanks for quick response, Andrew. That doesn't seem to work for me, but I'll try debugging more tomorrow!

[garethsb]

OK, after a merry dance trying to understand why that URLError was occurring in that code, we found it wasn't happening there at all. It's coming from ramlfications.parse right up front... See jdiegodcp/ramlfications#111.

[andrewbonney]

Well that's a pain! I wonder how hard that would be to resolve.

[garethsb]

I've just hacked ramlfications/loader.py to work (= get on to the next problem).

<        base_path = "file://" + base_path
---
>        base_path = "file:///" + base_path.replace('\\','/')

[andrewbonney]

Does adding the following at the top of Specification.py after the imports solve the problem?

# Work around ramlfications Windows compatibility issues
def _parse_json(self, jsonfile, base_path):
    """
    Parses JSON as well as resolves any `$ref`s, including references to
    local files and remote (HTTP/S) files.
    """
    base_path = os.path.abspath(base_path)
    if not base_path.endswith("/"):
        base_path = base_path + "/"
    if os.name == "nt":
        base_path = "file:///" + base_path.replace('\\', '/')
    else:
        base_path = "file://" + base_path

    with open(jsonfile, "r") as f:
        schema = jsonref.load(f, base_uri=base_path, jsonschema=True)
    return schema


try:
    ramlfications.loader._parse_json = _parse_json
except AttributeError:
    pass

[garethsb]

I don't think so. Is the RAMLLoader being constructed on the fly so that the above doesn't have any effect on the loader instance actually used?

[garethsb]

Ah, no, I see. I just needed to tweak your workaround like so:

> import jsonref
...
<     ramlfications.loader._parse_json = _parse_json
---
>     ramlfications.loader.RAMLLoader._parse_json = _parse_json

Great.

[andrewbonney]

Yes, was just about to say that. more haste less speed needed on my part.

I'll refactor this into a slightly more sensible place.