"Could not retrieve GCR's access token" when using Workload Identity

[Laski]

Hello! We're trying to push to GCR from a Gitlab CI/CD job running on GKE, using Workload Identity to provide the service account binding and giving the permissions to the corresponding IAM service account.

We can verify that the pod the job is running on is using the correct service account:

$ curl -H "Metadata-Flavor:Google" http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/email
projects/eryx-varios/serviceAccounts/gitlab-runner-service-account@eryx-varios.iam.gserviceaccount.com

And that the service account does have the needed permissions (by providing the credentials explictly through a JSON key):

$ cat $GITLAB_SERVICE_ACCOUNT_JSON | docker login -u _json_key --password-stdin https://gcr.io/
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
$ docker push "$CI_APPLICATION_REPOSITORY:$CI_APPLICATION_TAG"
The push refers to repository [gcr.io/eryx-varios/eryx-app/ci-pipeline-gcloud]
(...)
# push succeeds

However when we use the docker-credential-gcr, docker fails to push (unauthorized):

$ docker-credential-gcr configure-docker
Configuring default registries....
WARNING: A long list of credential helpers may cause delays running 'docker build'.
We recommend passing the registry names via the --registries flag for the specific registries you are using
Adding config for all GCR registries.
/root/.docker/config.json configured to use this credential helper for GCR registries

$ docker push "$CI_APPLICATION_REPOSITORY:$CI_APPLICATION_TAG"
The push refers to repository [gcr.io/eryx-varios/eryx-app/ci-pipeline-gcloud]
(...)
unauthorized: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication

Trying to verify credentials also fails:

$ echo "https://gcr.io/" | docker-credential-gcr get
docker-credential-gcr/helper: could not retrieve GCR's access token: metadata: GCE metadata "instance/service-accounts/default/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloud-platform" not defined

Also, I don't know if this is relevant when using Workload Identity, but the VM on GKE has all access scopes enabled.

We would really like to avoid having to pass the JSON keys explicitly. Any help is appreciated.