auth.secret_name provided but still fails with "authentication required" on Docker Hub private repo / cosign

[funkypenguin]

Hey folks!

I'm trying to use a cosign validator against my private Docker Hub repos, but I've not been able to get connaisseur to authenticate - so I get errors like this when testing with kubectl run:

Error from server: admission webhook "connaisseur-svc.connaisseur.svc" denied the request: Unexpected Cosign exception for image "docker.io/myord/curl:latest": Error: GET https://index.docker.io/v2/myorg/curl/manifests/latest: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:gergorg/curl Type:repository]]
main.go:46: error during command execution: GET https://index.docker.io/v2/myorg/curl/manifests/latest: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:myorg/curl Type:repository]]

My validator is defined like this:

  - name: default
    type: cosign  # or other supported validator (e.g. "cosign")
    # host: notary.docker.io # in case of notaryv1, configure the notary server to be used
    auth: 
      secret_name: registry-credentials  # reference a k8s secret in the form required by the validator type (check the docs)
    trust_roots:
    # # the `default` key is used if no key is specified in image policy
    - name: default
      key: |  # enter your public key below (my note : the one below is a dummy one used for dev)
        -----BEGIN PUBLIC KEY-----
        MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEizCL5gnS98PS929yPNo61x7MSuhs
        g4qfWtLxfaRQfy2YPhZ2//y+AeaGLU9Y98jQHyZqLON4F+9DFJqX2RbwFQ==
        -----END PUBLIC KEY-----
    #  # or (only for notaryv1 validator)
    #  username: myuser
    #  password: mypass

And my I've confirmed that the credentials are correctly mounted into the pod container at /app/connaisseur-config/default/.docker/config.json.

When I set LOGLEVEL=DEBUG, I see the following output from connaisseur:

[2022-03-07 01:52:23,831] INFO: {'message': 'Unexpected Cosign exception for image "docker.io/myorg/curl:latest": Error: GET https://index.docker.io/v2/myorg/curl/manifests/latest: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:myorg/curl Type:repository]]\nmain.go:46: error during command execution: GET https://index.docker.io/v2/myorg/curl/manifests/latest: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:myorg/curl Type:repository]]\n.', 'context': {'trust_data_type': 'dev.cosignproject.cosign/signature', 'stderr': 'Error: GET https://index.docker.io/v2/myorg/curl/manifests/latest: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:myorg/curl Type:repository]]\nmain.go:46: error during command execution: GET https://index.docker.io/v2/myorg/curl/manifests/latest: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:myorg/curl Type:repository]]\n', 'image': 'docker.io/myorg/curl:latest', 'trust_root': 'default', 'detection_mode': False}}
[2022-03-07 01:52:23,832] DEBUG: No alerting configuration file found.
[2022-03-07 01:52:23,832] ERROR: {'message': 'Unexpected Cosign exception for image "docker.io/myorg/curl:latest": Error: GET https://index.docker.io/v2/myorg/curl/manifests/latest: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:myorg/curl Type:repository]]\nmain.go:46: error during command execution: GET https://index.docker.io/v2/myorg/curl/manifests/latest: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:myorg/curl Type:repository]]\n.', 'context': {'trust_data_type': 'dev.cosignproject.cosign/signature', 'stderr': 'Error: GET https://index.docker.io/v2/myorg/curl/manifests/latest: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:myorg/curl Type:repository]]\nmain.go:46: error during command execution: GET https://index.docker.io/v2/myorg/curl/manifests/latest: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:myorg/curl Type:repository]]\n', 'image': 'docker.io/myorg/curl:latest', 'trust_root': 'default', 'detection_mode': False, 'user': 'system:admin', 'operation': 'CREATE', 'kind': 'Pod', 'name': 'test', 'namespace': 'sandbox'}}

Any ideas? :)

Thanks,
D

[xopham]

Hey @funkypenguin , have you followed the description on how to setup the secret and does have the typical dockerconfigjson form on the pod?
Otherwise, DockerHub tends to respond with unauthorized if the requested artifact does not exist. Are you certain that there is no typo or that the specific image with tag latest does actually exist? Can you pull that same image locally, with the dockerconfig json that is mounted into Connaisseur? Have you tested to verify with pure cosign locally?

[funkypenguin]

Hey @xopham!

Yes, I've followed both the documented instructions (creating a generic secret and then setting --type), and I've tried this format:

kubectl create -n connaisseur secret docker-registry registry-credentials \
      --docker-server=https://index.docker.io \
      --docker-username='{{ docker_hub_username }}' \
      --docker-password='{{ docker_hub_password }}'

In both cases, I get what looks like a valid /app/connaisseur-config/default/.docker/config.json in the pod (although the second format results in a file with no newlines).

To confirm whether the image is valid, here's what connaisseur says:

root@cn1:~# k run -n sandbox test --image myorg/curl:latest
Error from server: admission webhook "connaisseur-svc.connaisseur.svc" denied the request: Unexpected Cosign exception for image "docker.io/myorg/curl:latest": Error: GET https://index.docker.io/v2/myorg/curl/manifests/latest: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:myorg/curl Type:repository]]
main.go:46: error during command execution: GET https://index.docker.io/v2/myorg/curl/manifests/latest: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:myorg/curl Type:repository]]

Now I've not signed this image with cosign...

root@cn1:~# cosign verify myorg/curl:latest --key cosign.pub
Error: no matching signatures:

main.go:46: error during command execution: no matching signatures:

However, it is a valid image, since when I use an invalid image, I get an "entity not found" error out of cosign:

root@cn1:~# cosign verify myorg/curl:latestbogus --key cosign.pub
Error: entity not found in registry
main.go:46: error during command execution: entity not found in registry
root@cn1:~#

To rule out the possibility of this being a problem with just unsigned images, I signed the image...

root@cn1:~# cosign sign myorg/curl:latest --key cosign.key
Enter password for private key:
Pushing signature to: index.docker.io/myorg/curl
root@cn1:~#

And I can verify it with cosign:

root@cn1:~# cosign verify myorg/curl:latest --key cosign.pub

Verification for index.docker.io/myorg/curl:latest --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - The signatures were verified against the specified public key

[{"critical":{"identity":{"docker-reference":"index.docker.io/myorg/curl"},"image":{"docker-manifest-digest":"sha256:a10e1fe8956ae8137ba1cde262d4566b5a75ff89b923f97792f40732f777883f"},"type":"cosign container image signature"},"optional":null}]

But connaisseur still claims authentication is required:

root@cn1:~# k run -n sandbox test --image myorg/curl:latest
Error from server: admission webhook "connaisseur-svc.connaisseur.svc" denied the request: Unexpected Cosign exception for image "docker.io/myorg/curl:latest": Error: GET https://index.docker.io/v2/myorg/curl/manifests/latest: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:myorg/curl Type:repository]]
main.go:46: error during command execution: GET https://index.docker.io/v2/myorg/curl/manifests/latest: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:myorg/curl Type:repository]]
.

To confirm that the dockerconfigjson is correct, I compared the .docker/config.json on my host with /app/connaisseur-config/default/.docker/config.json in the pod:

root@cn1:~# sha256sum ~/.docker/config.json
6649d3e1ee53484c9b1c240128abc5c5ba3ab737397ac39bd02f09ab15650123  /root/.docker/config.json

root@cn1:~# k exec -n connaisseur connaisseur-deployment-64bcd657b8-7d2vl -it sha256sum /app/connaisseur-config/default/.docker/config.json
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
6649d3e1ee53484c9b1c240128abc5c5ba3ab737397ac39bd02f09ab15650123  /app/connaisseur-config/default/.docker/config.json
root@cn1:~#

[xopham]

See #575 . Fix is simple, but would need to reproduce the bug and check the fix as well.

[funkypenguin]

FWIW, I've confirmed that authentication works on v2.4.1, but interestingly.. not when using a kubernetes secret created with kubectl create -n connaisseur secret docker-registry, because (at first glance) this seems to create a .docker/config.json without newlines, which results in the same "authentication required" error, even on v2.4.1. When I instead create a generic secret (per the docs), with --type, then authentication works under v2.4.1

[xopham]

@funkypenguin interesting find 💯 could you open a separate issue for that?
as for the broken authentication, I have created a Pr #576 and hope we can release it soon.

[funkypenguin]

Re the authentication failure, it turned out to be the difference between docker.io with an implied /v2/ at the end, or docker.io/v1/. I'm still not sure exactly why it happens, but I appended my results to an existing issue here : sigstore/cosign#587 (comment)

[xopham]

The fix has just been released in v2.5.1. Thanks for raising this issue 🙏