Support Local Development with Kubernetes with Docker Desktop #3158

rgaiacs · 2025-01-07T17:05:32Z

No description provided.

rgaiacs · 2025-01-07T17:07:04Z

When I run

python deploy.py docker-desktop

I get the error

Error: UPGRADE FAILED: failed to create resource: Ingress.networking.k8s.io "static" is invalid: spec: Invalid value: []networking.IngressRule(nil): either `defaultBackend` or `rules` must be specified
Traceback (most recent call last):
  File "/home/raniere/mybinder.org-deploy/deploy.py", line 543, in <module>
    main()
  File "/home/raniere/mybinder.org-deploy/deploy.py", line 539, in main
    deploy(args.release, args.name, args.dry_run, args.diff)
  File "/home/raniere/mybinder.org-deploy/deploy.py", line 241, in deploy
    check_call(helm, dry_run)
  File "/home/raniere/mybinder.org-deploy/deploy.py", line 49, in check_call
    subprocess.check_call(cmd)
  File "/usr/lib/python3.10/subprocess.py", line 369, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['helm', 'upgrade', '--install', '--cleanup-on-fail', '--create-namespace', '--namespace=docker-desktop', 'docker-desktop', 'mybinder', '-f', 'secrets/config/common/bans.yaml', '-f', 'secrets/config/common/common.yaml', '-f', 'secrets/config/common/cryptnono.yaml', '-f', 'config/docker-desktop.yaml']' returned non-zero exit status 1.

@manics could you point me to the configuration that I'm missing? Thanks!

manics

Could it be

mybinder.org-deploy/config/staging.yaml

Lines 136 to 140 in 9a88e71

    
           static: 
        
             ingress: 
        
               hosts: 
        
                 - static.staging.mybinder.org 
        
                 - static.gke2.staging.mybinder.org

manics · 2025-01-07T18:24:51Z

deploy.py

@@ -437,6 +437,7 @@ def main():
        "release",
        help="Release to deploy",
        choices=[
+            "docker-desktop",


Can we rename this to local or something similar? I don't think this should be specific to docker-desktop, e.g. it should work with a local k3s/microk8s etc. It also means we could run this in GitHub CI using https://github.com/jupyterhub/action-k3s-helm which will allow us to test some changes in PRs instead of having to deploy to staging first.

rgaiacs · 2025-01-08T15:00:09Z

Thanks @manics for the feedback. I fixed the previously mentioned failed to create resource: Ingress.networking.k8s.io. I also re-wrote the documentation page to be more generic.

Following the provided documentation, the user should be able to

see the mybinder.org UI
start the creation of a new container image (for example, based on https://github.com/binder-examples/requirements)
start a container based on the newly created image
access the running container

At the moment, the user will get the following error

[I 250108 14:56:29 builder:669] Launching server for https://github.com/binder-examples/requirements: 0 other servers running this repo (0 total)
[I 250108 14:56:29 launcher:198] Creating user binder-examples-requirements-xwjx20bu for image binder-2dexamples-2drequirements-55ab5c:50533eb470ee6c24e872043d30b2fee463d6943f
[E 250108 14:56:49 launcher:123] Error accessing Hub API (using hub/api/users/binder-examples-requirements-xwjx20bu): HTTP 599: Connection timed out after 20001 milliseconds
[E 250108 14:57:13 launcher:123] Error accessing Hub API (using hub/api/users/binder-examples-requirements-xwjx20bu): HTTP 599: Connection timed out after 20000 milliseconds

This should be solved by changing /etc/hosts.

rgaiacs · 2025-01-08T15:30:13Z

I try using

binderhub:
  config:
    BinderHub:
      hub_url: http://jupyterhub.localhost

to fix the Error accessing Hub API but it did not work.

@manics do you have a suggestion?

manics · 2025-01-08T23:34:03Z

I don't think we should recommend modifying the hosts file, it should be possible to either rely on

kube-dns to resolve internal service names
a service like https://nip.io/ so that IP addresses 1.2.3.4 can be treated as a hostname 1.2.3.4.nip.io which is required for Kubernetes ingress

rgaiacs · 2025-01-09T10:15:48Z

In 2a6514a, the launch requests produces the following log in the binder pod

[I 250109 09:26:27 build:563] Started build build-binder-2dexamples-2drequirements-55ab5c-505-2c
[I 250109 09:26:27 build:565] Watching build pod build-binder-2dexamples-2drequirements-55ab5c-505-2c
[I 250109 09:26:29 build:647] Watching logs of build-binder-2dexamples-2drequirements-55ab5c-505-2c
[I 250109 09:26:36 build:679] Finished streaming logs of build-binder-2dexamples-2drequirements-55ab5c-505-2c
[I 250109 09:26:38 builder:669] Launching server for https://github.com/binder-examples/requirements: 0 other servers running this repo (0 total)
[I 250109 09:26:38 launcher:198] Creating user binder-examples-requirements-c4vstg48 for image binder-2dexamples-2drequirements-55ab5c:50533eb470ee6c24e872043d30b2fee463d6943f
[E 250109 09:26:58 launcher:123] Error accessing Hub API (using hub/api/users/binder-examples-requirements-c4vstg48): HTTP 599: Connection timed out after 20001 milliseconds

My understanding is that the binder pods tries to contact the hub pod but the request fails because hub_url is not defined.

Define hub_url to localhost or 127.0.0.1 fails because the binder pod will make a request to itself. The solution here is to use the "public" IP of the machine. In 3f76268, the user is required to provide their "public" IP and the launch requests produces the following log in the binder pod

[I 250109 10:06:13 build:563] Started build build-binder-2dexamples-2drequirements-55ab5c-505-2c
[I 250109 10:06:13 build:565] Watching build pod build-binder-2dexamples-2drequirements-55ab5c-505-2c
[I 250109 10:06:18 build:647] Watching logs of build-binder-2dexamples-2drequirements-55ab5c-505-2c
[I 250109 10:06:19 build:679] Finished streaming logs of build-binder-2dexamples-2drequirements-55ab5c-505-2c
[I 250109 10:06:21 builder:669] Launching server for https://github.com/binder-examples/requirements: 0 other servers running this repo (0 total)
[I 250109 10:06:21 launcher:198] Creating user binder-examples-requirements-guutvpki for image binder-2dexamples-2drequirements-55ab5c:50533eb470ee6c24e872043d30b2fee463d6943f
[I 250109 10:06:21 launcher:258] Starting server for user binder-examples-requirements-guutvpki with image binder-2dexamples-2drequirements-55ab5c:50533eb470ee6c24e872043d30b2fee463d6943f

The binder pod successfully contact the hub pod. Unfortunately, the user pod (jupyter-binder-examples-requirements-guutvpki) fails in the tc-init container with

+ tc qdisc add dev eth0 root handle 1: htb default 30
RTNETLINK answers: No such file or directory

Based on https://stackoverflow.com/a/16235349/1802726, the error is because of missing Linux kernel modules that I'm assuming are not available on Docker Desktop.

@manics does this looks good for you? Can we merge? I will try to test on my GNU/Linux machine next week.

rgaiacs · 2025-01-27T14:02:16Z

I try to reproduce the error

+ tc qdisc add dev eth0 root handle 1: htb default 30
RTNETLINK answers: No such file or directory

with the BinderHub helm instead of the mybinder.org helm when launching https://github.com/binder-examples/requirements but BinderHub helm does not have this error.

@manics @minrk @yuvipanda do you think this could be because of the system charts deployed?

manics · 2025-01-29T22:46:54Z

Which pod are you seeing the tc-init error in?

I'll try and review this PR in the next few days.

rgaiacs · 2025-01-30T09:42:27Z

The tc-init error happens on the user's jupyter- pod.

manics

Thanks for your work on this! I've made some suggestions, but otherwise I'm happy to see this merged and refinements made in later PRs. For example, I think it'd be interesting to enable some of the disabled services and test this in K3s in a GitHub workflow. It might also be interesting to see if we can use the configuration from @yuvipanda's new standalone 2i2c deployment in a GitHub workflow CI test. This would provide a first line of testing, instead of relying on a staging deployment to pick up errors.

manics · 2025-02-02T14:56:49Z

deploy.py

+    dry_run=False,
+    diff=False,
+    ip_address=None,
+    disable_ingress_nginx=False,


We may want to add further enables/disables. Instead of adding more flags in future do you think this could be additional_helm_args=None (or helm_args, or some other name), and we can pass a list of args instead? Or maybe additional_helm_set/helm_set to only allow --set arguments?

manics · 2025-02-02T15:01:49Z

deploy.py

+            ]
+        )
+
+    if disable_ingress_nginx:


If you're happy with the idea of additional_helm_args/additional_helm_set etc this can become either helm.extend(additional_helm_args) or helm.extend(f"--set={arg}" for arg in additional_helm_set)

manics · 2025-02-02T15:05:45Z

deploy.py

@@ -474,9 +507,20 @@ def main():
        default=stages[0],
        help="Stage to deploy, default all",
    )
+    argparser.add_argument(
+        "--disable-ingress-nginx",


If we add additional_helm_args or similar then

argparser.add_argument( "--helmarg", action="append", )

Should ensure that
deploy.py .... --helmarg='--set=ingress-nginx.enabled=false' --helmarg='--set=something.else=1'
results in args.helmarg == ["--set=ingress-nginx.enabled=false", "--set=something.else=1"].

manics · 2025-02-02T15:07:29Z

docs/source/getting_started/local_environment.md

@@ -0,0 +1,63 @@
+# Getting started with local development
+
+This page contains a starting point for people who want to know more about the BinderHub deployment by playing around with a local development instance.


Suggested change

This page contains a starting point for people who want to know more about the BinderHub deployment by playing around with a local development instance.

This page contains a starting point for people who want to know more about the mybinder.org deployment by playing around with a local development instance.

manics · 2025-02-02T15:08:33Z

docs/source/getting_started/local_environment.md

+
+You will need a local Kubernetes cluster. A few options are
+
+- [Kubernetes with Docker Desktop](https://docs.docker.com/desktop/features/kubernetes/) (recommended)


Suggested change

- [Kubernetes with Docker Desktop](https://docs.docker.com/desktop/features/kubernetes/) (recommended)

- [Kubernetes with Docker Desktop](https://docs.docker.com/desktop/features/kubernetes/)

Let's not recommend something with a complicated license.

Rancher Desktop is similar and open-source

manics · 2025-02-02T15:09:53Z

docs/source/getting_started/local_environment.md

+- [k3d](https://k3d.io/stable/)
+- [kind](https://kind.sigs.k8s.io/)
+
+### Install Docker Desktop


Do we need this section, or should it be obvious how to install the above applications?

manics · 2025-02-02T15:10:31Z

docs/source/getting_started/local_environment.md

+
+## Access your mybinder.org
+
+Open http://localhost with your favourite web browser.


Shouldn't this be http://xxx.xxx.xxx.xxx.nip.io?

Related to jupyterhub#3157

Relates to jupyterhub#3157

in localhost

Fixes File "/usr/local/lib/python3.11/site-packages/google/auth/_default.py", line 691, in default raise exceptions.DefaultCredentialsError(_CLOUD_SDK_MISSING_CREDENTIALS) google.auth.exceptions.DefaultCredentialsError: Your default credentials were not found. To set up Application Default Credentials, see https://cloud.google.com/docs/authentication/external/set-up-adc for more information.

for more information, see https://pre-commit.ci

for localhost.

and overwrite Helm chart with the provided IP address.

for more information, see https://pre-commit.ci

because Let's Encrypt will probably not work.

because it might already be installed in local environment

for more information, see https://pre-commit.ci

rgaiacs · 2025-03-10T11:53:08Z

@manics I addressed the suggestions.

rgaiacs self-assigned this Jan 7, 2025

manics reviewed Jan 7, 2025

View reviewed changes

rgaiacs force-pushed the 3157-kubernetes-with-docker-desktop branch 3 times, most recently from 57b9030 to 2a6514a Compare January 8, 2025 14:50

rgaiacs changed the title ~~WIP: Support Local Development with Kubernetes with Docker Desktop~~ Support Local Development with Kubernetes with Docker Desktop Jan 8, 2025

rgaiacs force-pushed the 3157-kubernetes-with-docker-desktop branch 2 times, most recently from 02887db to 109886d Compare January 27, 2025 13:19

manics reviewed Feb 2, 2025

View reviewed changes

manics mentioned this pull request Mar 10, 2025

Fix typo by remove extra - in docker call #3252

Merged

rgaiacs added 12 commits March 10, 2025 12:32

Start documentation to use Docker Desktop

76e8d9d

Related to jupyterhub#3157

Add Docker Desktop to deploy.py

eafe4be

Related to jupyterhub#3157

Expand notes for local development

fbd8547

Relates to jupyterhub#3157

Rename file in documentation

a0ffdf3

Support other local Kubernetes in addition to Docker Desktop

75ad659

Add ingress for static

f864b7e

in localhost

Disable analytics publisher

29c6683

Add call to charpress in documentation

624a313

Disable TLS for static in localhost

3d92a52

Update ingress hosts for localhost

db74338

Disable HTTPS for binderhub on localhost

c9828be

rgaiacs and others added 14 commits March 10, 2025 12:32

Add last section to documentation of localhost

0f57a3a

[pre-commit.ci] auto fixes from pre-commit.com hooks

06afe27

for more information, see https://pre-commit.ci

Add hub_url to binderhub.config.BinderHub

665b2e5

for localhost.

Force use of IP other than 127.0.0.1 for localhost

2579d9e

and overwrite Helm chart with the provided IP address.

[pre-commit.ci] auto fixes from pre-commit.com hooks

bc643fa

for more information, see https://pre-commit.ci

Remove section about container registry for local development

900f03c

Add --local-ip to documentation of local development

add1ee2

Add syntax highlight to code blocks

b768f2c

Remove certmanager from local deployment

be710ba

because Let's Encrypt will probably not work.

Add option --disable-ingress-nginx

a0d8411

because it might already be installed in local environment

Add prefix to nip.io

276883d

Review list of local kubernetes

3034efb

Fix URL to access local deployment

114ee79

Replace disable_ingress_nginx with additional_helm_args

baf3fce

rgaiacs force-pushed the 3157-kubernetes-with-docker-desktop branch from c7455d0 to baf3fce Compare March 10, 2025 11:50

[pre-commit.ci] auto fixes from pre-commit.com hooks

9fc8eee

for more information, see https://pre-commit.ci

rgaiacs requested a review from manics March 10, 2025 11:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support Local Development with Kubernetes with Docker Desktop #3158

Support Local Development with Kubernetes with Docker Desktop #3158

rgaiacs commented Jan 7, 2025

rgaiacs commented Jan 7, 2025

manics left a comment

manics Jan 7, 2025

rgaiacs commented Jan 8, 2025

rgaiacs commented Jan 8, 2025

manics commented Jan 8, 2025

rgaiacs commented Jan 9, 2025

rgaiacs commented Jan 27, 2025

manics commented Jan 29, 2025

rgaiacs commented Jan 30, 2025

manics left a comment •

edited

Loading

manics Feb 2, 2025

manics Feb 2, 2025

manics Feb 2, 2025

manics Feb 2, 2025

manics Feb 2, 2025

manics Feb 2, 2025

manics Feb 2, 2025

rgaiacs commented Mar 10, 2025

	static:
	ingress:
	hosts:
	- static.staging.mybinder.org
	- static.gke2.staging.mybinder.org

		@@ -0,0 +1,63 @@
		# Getting started with local development

		This page contains a starting point for people who want to know more about the BinderHub deployment by playing around with a local development instance.


		You will need a local Kubernetes cluster. A few options are

		- [Kubernetes with Docker Desktop](https://docs.docker.com/desktop/features/kubernetes/) (recommended)


		## Access your mybinder.org

		Open http://localhost with your favourite web browser.

Support Local Development with Kubernetes with Docker Desktop #3158

Are you sure you want to change the base?

Support Local Development with Kubernetes with Docker Desktop #3158

Conversation

rgaiacs commented Jan 7, 2025

rgaiacs commented Jan 7, 2025

manics left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rgaiacs commented Jan 8, 2025

rgaiacs commented Jan 8, 2025

manics commented Jan 8, 2025

rgaiacs commented Jan 9, 2025

rgaiacs commented Jan 27, 2025

manics commented Jan 29, 2025

rgaiacs commented Jan 30, 2025

manics left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rgaiacs commented Mar 10, 2025

manics left a comment •

edited

Loading