Connecting to kernel in a Docker image

[jkitchin]

I tried following the clues at #61, but I did not succeed.

First, I ran:

#+BEGIN_SRC sh
docker pull jupyter/base-notebook
docker run --rm --name test -p 56406-56410:56406-56410 \
    jupyter/base-notebook start.sh jupyter-kernel \
       --ip=0.0.0.0 \
       --KernelManager.control_port=56406 \
       --KernelManager.hb_port=56407 \
       --KernelManager.iopub_port=56408 \
       --KernelManager.shell_port=56409 \
       --KernelManager.stdin_port=56410
#+END_SRC

This creates a conn file:

#+BEGIN_SRC sh
docker exec -i test sh -c 'ls $(jupyter --runtime-dir)'
#+END_SRC

#+RESULTS:
: kernel-d34fab4c-5e95-4eb0-9692-1e0b81c28c3b.json

I use that in a session:

#+BEGIN_SRC jupyter-python :session /docker:test:/home/jovyan/.local/share/jupyter/runtime/kernel-d34fab4c-5e95-4eb0-9692-1e0b81c28c3b.json
1 + 3
#+END_SRC

But this times out:

Requesting kernel info...
or: Timeout before idle: #s(jupyter-org-request "3714e040-b459-4ad6-9f05-caa6cf45bbb7"

It seems like the ports in the kernelspec do not match what is exposed. I guess that is the problem?

#+BEGIN_SRC sh
docker exec -i test sh -c 'cat $(jupyter --runtime-dir)/*.json'
#+END_SRC

#+RESULTS:
| {                   |                                      |
| "shell_port":       | 34415,                               |
| "iopub_port":       | 35037,                               |
| "stdin_port":       | 37001,                               |
| "control_port":     | 46099,                               |
| "hb_port":          | 33903,                               |
| "ip":               | "0.0.0.0",                           |
| "key":              | "0d78349c-b9609d0321c4bb1441f8ac13", |
| "transport":        | "tcp",                               |
| "signature_scheme": | "hmac-sha256",                       |
| "kernel_name":      | python3                              |
| }                   |                                      |

I also tried copying this file to my local host like this:

#+BEGIN_SRC sh
docker exec -it test sh -c 'cp $(jupyter --runtime-dir)/*.json ~/conn.json'
docker cp test:/home/jovyan/conn.json ~/docker-test-conn.json
#+END_SRC

#+BEGIN_SRC jupyter-python :session ~/docker-test-conn.json
print("test")
#+END_SRC

but it also times out. I guess it is related to the ports? I checked these, and they are consistent.

#+BEGIN_SRC emacs-lisp
(jupyter-tunnel-connection "/docker:test:~/conn.json")
#+END_SRC

#+RESULTS:
| :shell_port | 34415 | :iopub_port | 35037 | :stdin_port | 37001 | :control_port | 46099 | :hb_port | 33903 | :ip | 0.0.0.0 | :key | 0d78349c-b9609d0321c4bb1441f8ac13 | :transport | tcp | :signature_scheme | hmac-sha256 | :kernel_name | python3 |

#+BEGIN_SRC emacs-lisp
(jupyter-read-plist  "/docker:test:~/conn.json")
#+END_SRC

#+RESULTS:
| :shell_port | 34415 | :iopub_port | 35037 | :stdin_port | 37001 | :control_port | 46099 | :hb_port | 33903 | :ip | 0.0.0.0 | :key | 0d78349c-b9609d0321c4bb1441f8ac13 | :transport | tcp | :signature_scheme | hmac-sha256 | :kernel_name | python3 |

#+BEGIN_SRC emacs-lisp
(and (file-remote-p "/docker:test:~/conn.json")
     (functionp 'tramp-dissect-file-name))
#+END_SRC

#+RESULTS:
: t

I am using Using elpa/jupyter-20230627.163019.

Any hints on how to make this work?

[pati-ni]

I am not sure how to make the exact existing workflow work. I remember having trouble passing the port information using the jupyter command, so this looks more of a jupyter issue. If you login to your docker you will see a kernel-....json file under the runtime directory. These are the ports that are being currently used.

As a workaround, I ended up invoking ipython in the remote machine.

The following workaround requires an ssh server in your docker. ssh is being used to tunnel your python session.

One way to make the kernel respect the predefined ports is creating a custom emacs.json file.

{
  "shell_port": 38363,
  "iopub_port": 41553,
  "stdin_port": 43343,
  "control_port": 33591,
  "hb_port": 40639,
  "ip": "127.0.0.1",
  "key": "cfde00e3-b2509ce23915d2d0af176b45",
  "transport": "tcp",
  "signature_scheme": "hmac-sha256",
  "kernel_name": ""
}

Place the file in the jupyter runtime location of your docker filesystem and then start a kernel using ipython instead on your docker:

ipython -f emacs.json

This will respect the ports specified in the json file.

Then on your local machine:

jupyter console --existing=./emacs.json --ssh docker

If the command succeeds you will see an ipython prompt [1]:

Also, it will create an emacs-ssh.json file which in turn use it in your emacs

With the latest emacs-jupyter this is what I need to get it working:

#+begin_src jupyter-python :kernel python :session ./emacs-ssh.json
!hostname
#+end_src

A bit clunky setup but it works ok and does not rely on tramp. I have managed to automate most of these steps with shell scripts.

[jkitchin]

what version of ipython do you use? I get:

[TerminalIPythonApp] WARNING | Unrecognized alias: 'f', it will have no effect.
Python 3.11.4 | packaged by conda-forge | (main, Jun 10 2023, 18:08:17) [GCC 12.2.0]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.14.0 -- An enhanced Interactive Python. Type '?' for help.

I guess you mean ipython kernel -f emacs.json?

That does work, but it creates an ssh file with different ports than the ones I specified.

This command seems to hang

jupyter console --existing=./emacs.json --ssh docker
[ZMQTerminalIPythonApp] Forwarding connections to 127.0.0.1 via docker
[ZMQTerminalIPythonApp] To connect another client via this tunnel, use:
[ZMQTerminalIPythonApp] --existing emacs-ssh.json

until I get

    raise RuntimeError("Kernel didn't respond to kernel_info_request") from e
RuntimeError: Kernel didn't respond to kernel_info_request

I am trying to use the Jupyter Stacks images, which maybe don't support ssh?

[pati-ni]

I guess you mean ipython kernel -f emacs.json?

Yes, you are right

I could reproduce the issue with that specific docker image. I am not sure why this is not working, the ports are unreachable. Also, ssh is not required since you can bind locally.

Sorry for the unhelpful comment.

[jkitchin]

Does it work for some other docker image that you use?

[pati-ni]

Does it work for some other docker image that you use?

No, I don't use docker containers but I do use kernels that sit in different network subnets. What I suggested works when I execute `ipython kernel -f emacs.json` locally and I am unsure why port forwarding is not working when this docker container is involved.

[jkitchin]

It looks like it might not possible to do this right now without patching jupyter.

jupyter/jupyter_client#955 (comment)

[pati-ni]

FYI, with what I suggested it looks like you need to expose the rest of the ports here to get it working:

https://github.com/jupyter/docker-stacks/blob/main/base-notebook/Dockerfile#L51

[jkitchin]

I have a solution of sorts. It looks like it is necessary to patch jupyter because of this issue: jupyter/jupyter_client#955 (comment).

Here are steps that led to a working solution for me.

First we modify jupyter so that the ports we specify get used, and not random ones. This is a somewhat fragile way to do this (e.g. the path to the file is Python version dependent(, but as a POC it is ok. It is crucial we start a kernel with these ports because otherwise there is no way to tell which ports to expose when running the image.

#+BEGIN_SRC text :tangle Dockerfile
FROM jupyter/scipy-notebook:python-3.11

RUN sed -i 's/    ports_cached = False/    ports_cached = True/g' /opt/conda/lib/python3.11/site-packages/jupyter_client/provisioning/local_provisioner.py 
#+END_SRC

Build the docker image.

#+BEGIN_SRC sh
docker build -t nocache .
#+END_SRC

Start the docker image with published ports and options to the kernel.

#+BEGIN_SRC sh
docker run --rm --name nocache -p 56406-56410:56406-56410 \
    nocache start.sh jupyter-kernel \
       --ip=0.0.0.0 \
       --KernelManager.control_port=56406 \
       --KernelManager.hb_port=56407 \
       --KernelManager.iopub_port=56408 \
       --KernelManager.shell_port=56409 \
       --KernelManager.stdin_port=56410 
#+END_SRC

We assume there is one of these connection files:

#+BEGIN_SRC sh
docker exec -i nocache sh -c 'ls $(jupyter --runtime-dir)'
#+END_SRC

#+RESULTS:
: kernel-d768b5b7-3085-4d15-b9de-15e66daee0c6.json

Now, copy that file locally.

#+BEGIN_SRC sh
CONNFILE=`docker exec -i nocache sh -c 'ls $(jupyter --runtime-dir)'`
docker cp nocache:/home/jovyan/.local/share/jupyter/runtime/${CONNFILE} .
#+END_SRC

Now use that kernel file as the :session arg

#+BEGIN_SRC jupyter-python :session (string-trim (shell-command-to-string "docker exec -i nocache sh -c 'ls $(jupyter --runtime-dir)'"))
! hostname
import sys
print(sys.executable)
#+END_SRC

#+RESULTS:
:RESULTS:
525b5ec4c226
/opt/conda/bin/python
:END:

Note there will be a residual json file when you are done. Maybe a kill-buffer hook could be used to get rid of this. I think it could be possible to wrap this in elisp, but for now I am not doing that. I want to explore an ssh solution first I think.