GESIS server fails when pushing the image

[rgaiacs]

I tested the GESIS server with the "empty" Git repository https://github.com/rgaiacs/mwe4mybinder.org and the user log shows

Waiting for build to start...
Picked Git content provider.
Cloning into '/tmp/repo2docker7j5zbjeh'...
HEAD is now at f1c2b5a Initial commit
Python version unspecified, using current default Python version 3.10. This will change in the future.Building conda environment for python=3.10
Using PythonBuildPack builder
#0 building with "default" instance using docker driver

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 4.85kB done
#1 DONE 0.0s

#2 [internal] load metadata for docker.io/library/buildpack-deps:jammy
#2 DONE 0.6s

#3 [ 1/32] FROM docker.io/library/buildpack-deps:jammy@sha256:1a63cc50ec6f4f45440121af59c47f78ca390607714ac0886c1588d118526b40
#3 DONE 0.0s

#4 [13/18] RUN chown jovyan:jovyan /home/jovyan
#4 CACHED

#5 [internal] load .dockerignore
#5 transferring context: 2B done
#5 DONE 0.0s

#6 [internal] load build context
#6 transferring context: 55.42kB 0.0s done
#6 DONE 0.0s

#7 [11/16] RUN if [ ! -d "/home/jovyan" ]; then         /usr/bin/install -o jovyan -g jovyan -d "/home/jovyan";     fi
#7 CACHED

#8 [ 7/16] COPY --chown=1000:1000 build_script_files/-2fopt-2fvenv-2flib-2fpython3-2e12-2fsite-2dpackages-2frepo2docker-2fbuildpacks-2fconda-2fenvironment-2epy-2d3-2e10-2dlinux-2d64-2elock-2bb6e8 /tmp/env/environment.lock
#8 CACHED

#9 [ 3/32] RUN echo "en_US.UTF-8 UTF-8" > /etc/locale.gen &&     locale-gen
#9 CACHED

#10 [ 9/16] RUN TIMEFORMAT='time: %3R' bash -c 'time /tmp/install-base-env.bash' && rm -rf /tmp/install-base-env.bash /tmp/env
#10 CACHED

#11 [13/16] RUN chown jovyan:jovyan /home/jovyan
#11 CACHED

#12 [ 4/32] RUN groupadd         --gid 1000         jovyan &&     useradd         --comment "Default user"         --create-home         --gid 1000         --no-log-init         --shell /bin/bash         --uid 1000         jovyan
#12 CACHED

#13 [ 5/32] RUN apt-get -qq update &&     apt-get -qq install --yes --no-install-recommends        gettext-base        less        unzip        > /dev/null &&     apt-get -qq purge &&     apt-get -qq clean &&     rm -rf /var/lib/apt/lists/*
#13 CACHED

#14 [12/16] WORKDIR /home/jovyan
#14 CACHED

#15 [ 8/16] COPY --chown=1000:1000 build_script_files/-2fopt-2fvenv-2flib-2fpython3-2e12-2fsite-2dpackages-2frepo2docker-2fbuildpacks-2fconda-2finstall-2dbase-2denv-2ebash-637204 /tmp/install-base-env.bash
#15 CACHED

#16 [ 2/32] RUN apt-get -qq update &&     apt-get -qq install --yes --no-install-recommends locales > /dev/null &&     apt-get -qq purge &&     apt-get -qq clean &&     rm -rf /var/lib/apt/lists/*
#16 CACHED

#17 [10/16] RUN mkdir -p /srv/npm && chown -R jovyan:jovyan /srv/npm
#17 CACHED

#18 [ 6/16] COPY --chown=1000:1000 build_script_files/-2fopt-2fvenv-2flib-2fpython3-2e12-2fsite-2dpackages-2frepo2docker-2fbuildpacks-2fconda-2factivate-2dconda-2esh-e67d51 /etc/profile.d/activate-conda.sh
#18 CACHED

#19 [14/16] COPY --chown=1000:1000 src/ /home/jovyan/
#19 DONE 0.0s

#20 [15/16] COPY /python3-login /usr/local/bin/python3-login
#20 DONE 0.0s

#21 [16/16] COPY /repo2docker-entrypoint /usr/local/bin/repo2docker-entrypoint
#21 DONE 0.1s

#22 exporting to image
#22 exporting layers 0.0s done
#22 writing image sha256:cf74014e1a659ce70af34eeb807f439089263ff61a87ab84dd71fadf9f7aee71 done
#22 naming to docker.io/gesiscss/binder-r2d-g5b5b759-rgaiacs-2dmwe4mybinder-2eorg-5c5529:f1c2b5a658c819212139ef8b82ad335055f6c41d done
#22 DONE 0.0s

#23 pushing gesiscss/binder-r2d-g5b5b759-rgaiacs-2dmwe4mybinder-2eorg-5c5529:f1c2b5a658c819212139ef8b82ad335055f6c41d with docker
#23 pushing layer 3fe195e5c769
#23 pushing layer 7416ce608053
#23 pushing layer 2d85a96be1ce
#23 pushing layer ae098c20d483
#23 pushing layer 5f70bf18a086
#23 pushing layer 45cf5b5745c2
#23 pushing layer 7e5d8dc46f59
#23 pushing layer cdc321148327
#23 pushing layer f57802063fea
#23 pushing layer 468f61133dd2
#23 pushing layer d3e42451921b
#23 pushing layer cff0c8b3a131
#23 pushing layer 0a72ce9f0ac6
#23 pushing layer 330a3443e078
#23 pushing layer a3e1a9f8dd1f
#23 pushing layer 3088d1636682
#23 pushing layer b7d4ab99ea66
#23 pushing layer 270a1170e7e3
#23 pushing layer 3fe195e5c769 0.9s done
#23 pushing layer 7416ce608053 0.9s done
#23 pushing layer 2d85a96be1ce 0.9s done
#23 pushing layer ae098c20d483 0.9s done
#23 pushing layer 5f70bf18a086 0.9s done
#23 pushing layer 45cf5b5745c2 0.9s done
#23 pushing layer 7e5d8dc46f59 0.9s done
#23 pushing layer cdc321148327 0.9s done
#23 pushing layer f57802063fea 0.9s done
#23 pushing layer 468f61133dd2 0.9s done
#23 pushing layer d3e42451921b 0.9s done
#23 pushing layer cff0c8b3a131 0.9s done
#23 pushing layer 0a72ce9f0ac6 0.9s done
#23 pushing layer 330a3443e078 0.9s done
#23 pushing layer a3e1a9f8dd1f 0.9s done
#23 pushing layer 3088d1636682 0.9s done
#23 pushing layer b7d4ab99ea66 0.9s done
#23 pushing layer 270a1170e7e3 0.9s done
#23 ERROR: denied: requested access to the resource is denied
------
 > pushing gesiscss/binder-r2d-g5b5b759-rgaiacs-2dmwe4mybinder-2eorg-5c5529:f1c2b5a658c819212139ef8b82ad335055f6c41d with docker:
------
ERROR: denied: requested access to the resource is denied
Error during build: Command '['docker', 'buildx', 'build', '--progress', 'plain', '--push', '--build-arg', 'NB_USER=jovyan', '--build-arg', 'NB_UID=1000', '--tag', 'gesiscss/binder-r2d-g5b5b759-rgaiacs-2dmwe4mybinder-2eorg-5c5529:f1c2b5a658c819212139ef8b82ad335055f6c41d', '--platform', 'linux/amd64', '/tmp/tmpzmpw3wvj']' returned non-zero exit status 1.

This is happening after #3225 when mybinder.org started using docker buildx. My understanding of the error is that the credentials to the container registry used by GESIS are incorrect. But they were not changed.

@yuvipanda could you point to me where to configure the credentials used by docker buildx? Thanks!

cc @arnim

[yuvipanda]

@rgaiacs if you look under secrets/config/hetzner-2i2c.yaml, you'll see them under binderhub.registry and binderhub.jupyterhub.imagePullSecret

[yuvipanda]

Temporarily taken #3254 out of rotation while we fix this. I hadn't fully realized that the CI/CD tests won't catch GESIS failures on this repo. Sorry @rgaiacs

[yuvipanda]

i was also looking at grafana, and didn't spot the gesis failures there. will need to look into that as well

[rgaiacs]

you'll see them under binderhub.registry

No change on GESIS configuration. We have

binderhub:
  registry:
    username: ***
    password: ***

I looked at the Docker Hub account that GESIS is using and the last time that we was able to push an image was 5 days ago (6 Mar 2025 at 7:46 pm UTC+1).

On 6 Mar 2025, we updated the GESIS deployment to sync the KubernetesBuildExecutor.build_image. No other changes.

@yuvipanda do you know if buildx is configured to work only with access tokens instead of password?

Another possible cause for this problem is the changes announced at Revisiting Docker Hub Policies: Prioritizing Developer Experience.

Temporarily taken #3254 out of rotation

👍 Good call.

was also looking at grafana, and didn't spot the gesis failures there.

It is hard to spot because it only fails when building a new image. GESIS is operating a "perpetual" container register on Docker Hub and we pull existing images.

[rgaiacs]

I tested a local build and push from my laptop using the credentials from GESIS and it worked. The Dockerfile was

FROM alpine:3.21.3

WORKDIR /mnt/test

COPY ./README.md /mnt/test/README.md

and running

docker buildx build --progress plain --push --tag gesiscss/mwe-3253:1 .

returned

#0 building with "default" instance using docker driver

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 112B done
#1 DONE 0.0s

#2 [internal] load metadata for docker.io/library/alpine:3.21.3
#2 ...

#3 [auth] library/alpine:pull token for registry-1.docker.io
#3 DONE 0.0s

#2 [internal] load metadata for docker.io/library/alpine:3.21.3
#2 DONE 1.2s

#4 [internal] load .dockerignore
#4 transferring context: 2B done
#4 DONE 0.0s

#5 [1/3] FROM docker.io/library/alpine:3.21.3@sha256:a8560b36e8b8210634f77d9f7f9efd7ffa463e380b75e2e74aff4511df3ef88c
#5 resolve docker.io/library/alpine:3.21.3@sha256:a8560b36e8b8210634f77d9f7f9efd7ffa463e380b75e2e74aff4511df3ef88c 0.0s done
#5 CACHED

#6 [internal] load build context
#6 transferring context: 37B done
#6 DONE 0.0s

#7 [2/3] WORKDIR /mnt/test
#7 DONE 0.0s

#8 [3/3] COPY ./README.md /mnt/test/README.md
#8 DONE 0.0s

#9 exporting to image
#9 exporting layers 0.1s done
#9 exporting manifest sha256:d99bff2a38eded1ec1ffa8116720d0bb01e9b73c06c10a8fd910841bded2319c
#9 exporting manifest sha256:d99bff2a38eded1ec1ffa8116720d0bb01e9b73c06c10a8fd910841bded2319c 0.0s done
#9 exporting config sha256:bd8f7902d13d3061b30f43d7129136c262185d2d35e1b5b94f5bf05c4239ba9d 0.0s done
#9 exporting attestation manifest sha256:91ec0c09a52322d3e56c835022f7792c49ab77cc876f87fa66918e18f7be0fc2 0.0s done
#9 exporting manifest list sha256:6e7d45a28a8f7a82d0799a617c2f825ceadb7cd141a249726bef07f2a339386a 0.0s done
#9 naming to docker.io/gesiscss/mwe-3253:1 done
#9 unpacking to docker.io/gesiscss/mwe-3253:1 0.0s done
#9 pushing layers
#9 ...

#10 [auth] gesiscss/mwe-3253:pull,push token for registry-1.docker.io
#10 DONE 0.0s

#9 exporting to image
#9 pushing layers 7.4s done
#9 pushing manifest for docker.io/gesiscss/mwe-3253:1@sha256:6e7d45a28a8f7a82d0799a617c2f825ceadb7cd141a249726bef07f2a339386a
#9 pushing manifest for docker.io/gesiscss/mwe-3253:1@sha256:6e7d45a28a8f7a82d0799a617c2f825ceadb7cd141a249726bef07f2a339386a 1.7s done
#9 DONE 9.8s

#11 pushing gesiscss/mwe-3253:1 with docker
#11 pushing layer 108297fb44bf
#11 pushing layer dc672d261718 0.1s
#11 pushing layer f18232174bc9 0.1s
#11 pushing layer f1b6a732c212 0.1s
#11 pushing layer 108297fb44bf 2.9s done
#11 pushing layer dc672d261718 2.9s done
#11 pushing layer f18232174bc9 2.9s done
#11 pushing layer f1b6a732c212 2.9s done
#11 DONE 3.2s

But I tested from the GESIS server and I got the same error.

#22 pushing layer 270a1170e7e3 0.7s done
#22 ERROR: denied: requested access to the resource is denied
------
 > pushing gesiscss/binder-r2d-g5b5b759-rgaiacs-2dmwe4mybinder-2eorg-5c5529:6e8f31e93fd476581c07bc6a8bf407176033424d with docker:
------
ERROR: denied: requested access to the resource is denied
Error during build: Command '['docker', 'buildx', 'build', '--progress', 'plain', '--push', '--build-arg', 'NB_USER=jovyan', '--build-arg', 'NB_UID=1000', '--tag', 'gesiscss/binder-r2d-g5b5b759-rgaiacs-2dmwe4mybinder-2eorg-5c5529:6e8f31e93fd476581c07bc6a8bf407176033424d', '--platform', 'linux/amd64', '/tmp/tmpmzekq582']' returned non-zero exit status 1.Error in event stream: Error

@yuvipanda do you know a easy way to start a debugging build pod on Kubernetes with the same configuration?

[rgaiacs]

The good news is that I was able to replicate the error locally.

I used the dirty trick to provide a postBuild with

#!/bin/bash

tail -f /dev/null

This will keep the build pod on the server running and allowing me to inspect and debug it.

I copied the config.json provided by the secret build-docker-config to another machine (bare metal Linux) and run

docker buildx build --progress plain --push --tag gesiscss/mwe-3253:1 .

This time, I got the error

ERROR: denied: requested access to the resource is denied

After some reading and ChatGPT prompt, my understanding is that Docker Hub finally deprecated authentication with base64. On 14 August 2024, Docker announced Deprecation of Password Logins on CLI with Docker SSO Enforcement. But they didn't change the settings for other users at that time. The docker cli reference manual includes the following section:

Provide a password using STDIN (--password-stdin)

To run the docker login command non-interactively, you can set the --password-stdin flag to provide a password through STDIN. Using STDIN prevents the password from ending up in the shell's history, or log-files.

The following example reads a password from a file, and passes it to the docker login command using STDIN:

cat ~/my_password.txt | docker login --username foo --password-stdin

This will require changes on BinderHub.

[rgaiacs]

Please follow up jupyterhub/binderhub#1943.

[rgaiacs]

The problem is caused by a missing trailing slash in config.json, see jupyterhub/binderhub#1945 for a fix.

I'm trying to find if this change is on docker client.

From	Docker version	build	docker login generates config.json with trailing slash
	28.0.1	068a01e	✔
Ubuntu 22.04.4 LTS	27.5.1	a187fa5	✔
mybinder.org	27.3.1	ce1223035ac3ab8922717092e63a184cf67b493d

[yuvipanda]

We now do use docker login with --password-stdin for login: https://github.com/jupyterhub/repo2docker/blob/main/repo2docker/docker.py#L200. That article seems to imply this is only true for accounts that enforce SSO - is the gesis org enforcing SSO? I tried this locally with my yuvipanda account and it seems to work fine.

fwiw, this is also why I avoid using any LLM tools while debugging - they often lead me down wrong paths but they don't really have any way to know it's wrong...