Jupyter notebook on theia-python running on internal domain

[mridup]

following #423 :
we were able to correctly deploy theia-python docker on an Azure ACI instance with letsencrypt certificate and have the webviews and jupyter notebooks working correctly.

Further we deployed the theia-python docker on an internal domain with it's own CA certificate issued by internal Root CA.
So this domain cannot be reached from outside and also there is no outside connectivity from this host (except can be configured by corporate proxy of course).
So we solved the issue of web workers not running without SSL and also by configuring the THEIA_WEBVIEW_EXTERNAL_ENDPOINT as {{hostname}}

But now in the internal server, we are again not able to run jupyter notebook. There is no error related to webviews or web workers now and hence the previous issue in #423 is not the concern here.
The notebook tab does open but there is no rendering of the notebook cells etc. There is only one indication but not sure if it is related.
When the notebook is opened the output on browser console is the following:

Also in the docker logs there is a lot less logs output (expecially related to 'Python Extension') compared to the working case on Azure ACI. The logs does not indicate any direct reference to any error related to the python notebook.
To note here is that the images used for the internal server and Azure ACI are exactly the same except that the certificates and key are different in both cases.
Another difference is that the internal server container runs as root wherease the Azure ACI container runs as USER 'theia'. although this shouldn't make any difference.

Here are the docker logs from internal server domain when a python notebook is opened (does not work):
internal_server_logs.txt

And here are the docker logs from Azure ACI when a python notebook is opened (works):
azure_aci.txt

any pointers would be super helpful!

[marcdumais-work]

Thanks @mridup for trying this and sharing your results. I see a few exceptions in the azure log, that look interesting, that could potentially be related to the issue, or lead us towards it. See below.

FWIW, I also get a 404 on this URL: https://raw.githubusercontent.com/microsoft/vscode-python/master/experiments.json, I am not sure what the consequences are, when such a failure occurs, that causes the "experiments" to fail to initialize.

root ERROR [hosted-plugin: 749] Error Python Extension: 2020-10-25 06:44:19: Failed to download and store experiments, Class name = S, completed in 685ms, , Return Value: undefined Error: Failed with status 404, Not Found, Uri https://raw.githubusercontent.com/microsoft/vscode-python/master/experiments.json
    at H._callback (/home/theia/plugins/vscode-python/extension/out/client/extension.js:75:636745)
    at H.a._callback.a.callback.a.callback (/home/theia/plugins/vscode-python/extension/out/client/node_modules/request.js:195:5429)
    at H.emit (events.js:315:20)
    at H.<anonymous> (/home/theia/plugins/vscode-python/extension/out/client/node_modules/request.js:195:18242)
    at H.emit (events.js:315:20)
    at IncomingMessage.<anonymous> (/home/theia/plugins/vscode-python/extension/out/client/node_modules/request.js:195:17084)
    at Object.onceWrapper (events.js:421:28)
    at IncomingMessage.emit (events.js:327:22)
    at endReadableNT (_stream_readable.js:1220:12)
    at processTicksAndRejections (internal/process/task_queues.js:84:21)
root ERROR [hosted-plugin: 749] Error Python Extension: 2020-10-25 06:44:19: Failed to initialize experiments, Class name = S, completed in 687ms, , Return Value: undefined Error: Failed with status 404, Not Found, Uri https://raw.githubusercontent.com/microsoft/vscode-python/master/experiments.json
    at H._callback (/home/theia/plugins/vscode-python/extension/out/client/extension.js:75:636745)
    at H.a._callback.a.callback.a.callback (/home/theia/plugins/vscode-python/extension/out/client/node_modules/request.js:195:5429)
    at H.emit (events.js:315:20)
    at H.<anonymous> (/home/theia/plugins/vscode-python/extension/out/client/node_modules/request.js:195:18242)
    at H.emit (events.js:315:20)
    at IncomingMessage.<anonymous> (/home/theia/plugins/vscode-python/extension/out/client/node_modules/request.js:195:17084)
    at Object.onceWrapper (events.js:421:28)
    at IncomingMessage.emit (events.js:327:22)
    at endReadableNT (_stream_readable.js:1220:12)
    at processTicksAndRejections (internal/process/task_queues.js:84:21)
root ERROR Failed to parse data from " /home/theia/.theia/plugin-storage/global-state.json ". Reason: SyntaxError: /home/theia/.theia/plugin-storage/global-state.json: Unexpected end of JSON input
    at JSON.parse (<anonymous>)
    at /home/theia/node_modules/jsonfile/index.js:33:18
    at /home/theia/node_modules/graceful-fs/graceful-fs.js:123:16
    at FSReqCallback.readFileAfterClose [as oncomplete] (internal/fs/read_file_context.js:63:3)
root ERROR Failed to parse data from " /home/theia/.theia/plugin-storage/global-state.json ". Reason: SyntaxError: /home/theia/.theia/plugin-storage/global-state.json: Unexpected end of JSON input
    at JSON.parse (<anonymous>)
    at /home/theia/node_modules/jsonfile/index.js:33:18
    at /home/theia/node_modules/graceful-fs/graceful-fs.js:123:16
    at FSReqCallback.readFileAfterClose [as oncomplete] (internal/fs/read_file_context.js:63:3)
root ERROR Failed to parse data from " /home/theia/.theia/plugin-storage/global-state.json ". Reason: SyntaxError: /home/theia/.theia/plugin-storage/global-state.json: Unexpected end of JSON input
    at JSON.parse (<anonymous>)
    at /home/theia/node_modules/jsonfile/index.js:33:18
    at /home/theia/node_modules/graceful-fs/graceful-fs.js:123:16
    at FSReqCallback.readFileAfterClose [as oncomplete] (internal/fs/read_file_context.js:63:3)
root ERROR Failed to parse data from " /home/theia/.theia/plugin-storage/global-state.json ". Reason: SyntaxError: /home/theia/.theia/plugin-storage/global-state.json: Unexpected end of JSON input
    at JSON.parse (<anonymous>)
    at /home/theia/node_modules/jsonfile/index.js:33:18
    at /home/theia/node_modules/graceful-fs/graceful-fs.js:123:16
    at FSReqCallback.readFileAfterClose [as oncomplete] (internal/fs/read_file_context.js:63:3)

root ERROR [hosted-plugin: 749] Error Python Extension: 2020-10-25 06:44:20: Failed to check if file needs to be fixed [Error: ENOENT: no such file or directory, open '/home/theia/.config/Code/User/settings.json'] {
  errno: -2,
  code: 'ENOENT',
  syscall: 'open',
  path: '/home/theia/.config/Code/User/settings.json'
}
root ERROR [hosted-plugin: 749] Error Python Extension: 2020-10-25 06:44:20: Failed to check if file needs to be fixed [Error: ENOENT: no such file or directory, open '/home/project/.vscode/settings.json'] {
  errno: -2,
  code: 'ENOENT',
  syscall: 'open',
  path: '/home/project/.vscode/settings.json'
}

root ERROR Failed to parse data from " /home/theia/.theia/workspace-storage/9c491c7c323deca0a9a4af50687fc724/workspace-state.json ". Reason: SyntaxError: /home/theia/.theia/workspace-storage/9c491c7c323deca0a9a4af50687fc724/workspace-state.json: Unexpected end of JSON input
    at JSON.parse (<anonymous>)
    at /home/theia/node_modules/jsonfile/index.js:33:18
    at /home/theia/node_modules/graceful-fs/graceful-fs.js:123:16
    at FSReqCallback.readFileAfterClose [as oncomplete] (internal/fs/read_file_context.js:63:3)

This one looks related to vscode-python telemetry - probably ok that it fails:

root ERROR [hosted-plugin: 749] Error Python Extension: 2020-10-25 06:44:19: Failed to serialize gatherRules for DATASCIENCE.SETTINGS TypeError: Cannot convert object to primitive value
    at Array.join (<anonymous>)
    at Array.toString (<anonymous>)
    at /home/theia/plugins/vscode-python/extension/out/client/extension.js:1:12839
    at Array.forEach (<anonymous>)
    at Object.l [as sendTelemetryEvent] (/home/theia/plugins/vscode-python/extension/out/client/extension.js:1:12756)
    at C.sendSettingsTelemetry (/home/theia/plugins/vscode-python/extension/out/client/extension.js:75:707093)
    at C.r.value (/home/theia/plugins/vscode-python/extension/out/client/extension.js:1:87450)
    at Timeout._onTimeout (/home/theia/plugins/vscode-python/extension/out/client/extension.js:1:85969)
    at listOnTimeout (internal/timers.js:549:17)
    at processTimers (internal/timers.js:492:7)

[marcdumais-work]

The first exception might have been fixed upstream a couple of months ago: microsoft/vscode-python#13839

[mridup]

The issue with the internal server comes when making a HTTP GET request and the request(service worker?) either remains pending or response is curtailed as depicted here:

and the working case with azure server:

Any ideas why this request would stall?

[mridup]

the above of course then goes on to trigger the "Syntax Error: Invalid or unexpected token" error we see in the console.