feat: Publish debug container image and account-sync sidecar #251
Conversation
account-sync/Dockerfile
Outdated
| @@ -0,0 +1,10 @@ | |||
| ARG PYTHON_VERSION=3.11 | |||
There was a problem hiding this comment.
I'm not attached to having this account-sync container built from this repository, but given the recent drop of Debian support for the nss-pam-ldap library in favour of nss-pam-ldapd, and as the configuration is so nestled deep in the inner workings of the container, this seemed a reasonable place for it to be built from to ensure it used a compatible version of the base container image.
There was a problem hiding this comment.
We could also just have a standalone repo in DiamondLightSource for this. It feels like the cadence of release of account-sync vs python-copier-template would be wildly different. Account sync could be built occasionally (almost never after the first time?) and if you update it then you need to update the version tag that copier template refers to.
There was a problem hiding this comment.
I think I prefer that, because otherwise there will be many releases of the account-sync image with the same contents.
There was a problem hiding this comment.
I'll probably end up moving it to its own repository and using some release automation that I want to try out to keep it up to date.
| @@ -14,10 +14,31 @@ ENV PATH=/venv/bin:$PATH{% if docker %} | |||
|
|
|||
| # The build stage installs the context into the venv | |||
| FROM developer AS build | |||
| COPY . /context | |||
| WORKDIR /context | |||
| # Requires buildkit 0.17.0 | |||
There was a problem hiding this comment.
To use chmod with non-octal form
|  | ||
| The Kubernetes plugin can be found in the plugin bar. Expanding the Clusters>`cluster`>Workloads>Pods views, your service should be visible. Right Click>Attach Visual Studio Code will initiate connecting to the workspace in the cluster. Select your service container from the top menu when prompted. | ||
|
|
||
| After the connection to the cluster has been established, it may be necessary to open the workspace folder by clicking the Explorer option in the plugin bar, the repository will be mounted at `/workspaces/<service name>`, equivalent to when working with a local devcontainer. |
There was a problem hiding this comment.
This does appear to always be necessary: container config files are supported for attaching to generic remote devcontainers, but not currently for those in kubernetes.
There was a problem hiding this comment.
Re kubeconfig. You can select your kubeconfig after launching vscode, see image. The important thing is that kubectl and helm are available on the path. This is challenging when vscode may be launched from gnome (or be already running and attach to the existing process). For this reason I suggest we recommend putting the (current version) tools in the path in .profile like I suggest for docker-compose here https://epics-containers.github.io/main/how-to/compose-quickstart.html#diamond-light-source-workstation. Note that this is peculiar to sites which use 'module load' to get these tools and they are not in the path by default.
| ``` | ||
|
|
||
| The following changes/additions to your `values.yaml` may be required to connect vscode when using the sidecar. | ||
| It is recommended to set the `HOME` environment variable on your container to be debugged to the same value used in the volume below. |
There was a problem hiding this comment.
From my experience, HOME has been unset, so trying to reference ~, or ~/.ssh routes to / or /.ssh respectively. My example-service has HOME being set if debug is enabled.
account-sync/Dockerfile
Outdated
| @@ -0,0 +1,10 @@ | |||
| ARG PYTHON_VERSION=3.11 | |||
There was a problem hiding this comment.
We could also just have a standalone repo in DiamondLightSource for this. It feels like the cadence of release of account-sync vs python-copier-template would be wildly different. Account sync could be built occasionally (almost never after the first time?) and if you update it then you need to update the version tag that copier template refers to.
account-sync/Dockerfile
Outdated
| @@ -0,0 +1,10 @@ | |||
| ARG PYTHON_VERSION=3.11 | |||
There was a problem hiding this comment.
I think I prefer that, because otherwise there will be many releases of the account-sync image with the same contents.
|  | ||
| The Kubernetes plugin can be found in the plugin bar. Expanding the Clusters>`cluster`>Workloads>Pods views, your service should be visible. Right Click>Attach Visual Studio Code will initiate connecting to the workspace in the cluster. Select your service container from the top menu when prompted. | ||
|
|
||
| After the connection to the cluster has been established, it may be necessary to open the workspace folder by clicking the Explorer option in the plugin bar, the repository will be mounted at `/workspaces/<service name>`, equivalent to when working with a local devcontainer. |
There was a problem hiding this comment.
Re kubeconfig. You can select your kubeconfig after launching vscode, see image. The important thing is that kubectl and helm are available on the path. This is challenging when vscode may be launched from gnome (or be already running and attach to the existing process). For this reason I suggest we recommend putting the (current version) tools in the path in .profile like I suggest for docker-compose here https://epics-containers.github.io/main/how-to/compose-quickstart.html#diamond-light-source-workstation. Note that this is peculiar to sites which use 'module load' to get these tools and they are not in the path by default.
| COPY . /context | ||
| WORKDIR /context | ||
| # Requires buildkit 0.17.0 | ||
| COPY --chmod=o+wrX . /workspaces/{{ repo_name }} |
There was a problem hiding this comment.
Do we want to make this all writable for developer as well as debug? It might not matter that much I guess?
|
Overall this is a neat solution - great work.
Other than that my comments were minor. |
DiamondJoseph
changed the title
DiamondJoseph
changed the title
There was a problem hiding this comment.
Thanks, this seems like a good solution to do something quite complicated. A few comments
- As we had a few mysterious issues when trying to get it to work for me, I don't think we can trust the steps in the docs to reliably "just work" yet.
- The details of how this works is pretty confusing for a kube noob, although conceptually "swapping out one container for another and attaching vscode" does make sense
- I agree with @gilesknap 's comment about adding a full helm chart example in the copier template
- To prove this is robust, we should do an iterative process of choosing someone in daq/controls and asking them to try and get this working using only the docs, then adjusting the docs with whatever they were missing
| With the [Kubernetes plugin for vscode](https://marketplace.visualstudio.com/items?itemName=ms-kubernetes-tools.vscode-kubernetes-tools) it is then possible to attach to the container inside the cluster. This may require that your targeted kubeconfig is at `~/.kube/config`, rather than referenced from the environment variable `KUBECONFIG`. It may also be necessary to [add additional contextual information](https://kubernetes.io/docs/reference/kubectl/generated/kubectl_config/kubectl_config_set-context/), such as the namespace. | ||
|
|
||
|  | ||
| The Kubernetes plugin can be found in the plugin bar. Expanding the Clusters>`cluster`>Workloads>Pods views, your service should be visible. Right Click>Attach Visual Studio Code will initiate connecting to the workspace in the cluster. Select your service container from the top menu when prompted. |
There was a problem hiding this comment.
Might be worth mentioning that you need to do a module load {k8s-cluster} in the terminal that you launch vscode from in order for the cluster to appear
|
|
||
| The container build also publishes a debug container for each tagged release of the container with the tag suffixed with `-debug`. This container contains the workspace and has an alternative entrypoint which allows the devcontainer to attach: if you have configured a `livenessProbe` that requires the service to have started it should be disabled. The container also installs debugpy and makes the service install editable. Any custom `command` or `args` defined for the container should be disabled. | ||
|
|
||
| With the [Kubernetes plugin for vscode](https://marketplace.visualstudio.com/items?itemName=ms-kubernetes-tools.vscode-kubernetes-tools) it is then possible to attach to the container inside the cluster. This may require that your targeted kubeconfig is at `~/.kube/config`, rather than referenced from the environment variable `KUBECONFIG`. It may also be necessary to [add additional contextual information](https://kubernetes.io/docs/reference/kubectl/generated/kubectl_config/kubectl_config_set-context/), such as the namespace. |
There was a problem hiding this comment.
Please could you add:
- How do we know if we need to change targeted kubeconfig
- How do we know if its necessary to do the 'add additional contextual information' bit
|
FYI I'm testing this PR against PandaBlocks-ioc. If anyone else wants to try it see the debug container at https://github.com/gilesknap/PandABlocks-ioc/pkgs/container/pandablocks-ioc/406412377?tag=1.0.0b1-debug |
So this works for me. For a python] IOC I needed to change the values.yaml as below. Use of root is being discouraged by the cloud team and may be switched off. But using this meant I did not need the sidecar. My limits needed updating or there was not enough memory for vscode server. # yaml-language-server: $schema=https://github.com/epics-containers/ec-helm-charts/releases/download/3.4.4/ioc-instance.schema.json#/$defs/service
ioc-instance:
#image: ghcr.io/pandablocks/pandablocks-ioc:0.11.2
image: ghcr.io/gilesknap/pandablocks-ioc:1.0.0b1-debug
iocConfig: /epics/ioc
# don't make the IOC the entrypoint
startCommand: sleep
startArgs: infinity
# use root for now
securityContext:
runAsUser: 0
runAsGroup: 0
# generous resources to handle the extra overheads of debugging
resources:
limits:
cpu: 4
memory: 4Gi
requests:
cpu: 1
memory: 1Gi |
Closes #234
Provides an account-sync sidecar for current Python base container image to allow vscode to connect to devcontainers running in the cluster as specific users.
Adds a stage to the container build to make a persistent devcontainer with editable code for connecting to.
Adds docs explaining how to configure and connect to containers running in the cluster.
I have made a test repository with Helm configuration and an example service which can be fixed internally to the cluster. I have a PR with commits that I made inside a devcontainer inside the cluster.