k8s connaisseur-deployment pods goes down immediately

[prasanjit-enginprogam]

Hello All,
k8s pods are going down within a couple of seconds once connaisseur-install-webhook-job and connaisseur-bootstrap-sentinel pods are completed.

I am using cosign (latest) and is following the official document

I used the main branch for helm deployment.

Versions:
connaisseur : v2.1.2
k8s : Major:"1", Minor:"20+", GitVersion:"v1.20.7-eks-d88609"

Output:

NAME                                          READY   STATUS    RESTARTS   AGE     IP              NODE                                          NOMINATED NODE   READINESS GATES
pod/connaisseur-deployment-6db7c98c8d-b42p5   0/1     Running   0          3m17s   10.150.21.85    ip-10-150-21-60.us-west-2.compute.internal    <none>           <none>
pod/connaisseur-deployment-6db7c98c8d-jznpd   0/1     Running   0          3m17s   10.150.14.237   ip-10-150-14-222.us-west-2.compute.internal   <none>           <none>
pod/connaisseur-deployment-6db7c98c8d-kstsk   0/1     Running   0          3m17s   10.150.44.135   ip-10-150-41-22.us-west-2.compute.internal    <none>           <none>

NAME                      TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE     SELECTOR
service/connaisseur-svc   ClusterIP   172.20.225.168   <none>        443/TCP   3m17s   app.kubernetes.io/instance=connaisseur,app.kubernetes.io/name=connaisseur

NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE     CONTAINERS    IMAGES                                        SELECTOR
deployment.apps/connaisseur-deployment   0/3     3            0           3m17s   connaisseur   securesystemsengineering/connaisseur:v2.1.2   app.kubernetes.io/instance=connaisseur,app.kubernetes.io/name=conn
aisseur

NAME                                                DESIRED   CURRENT   READY   AGE     CONTAINERS    IMAGES                                        SELECTOR
replicaset.apps/connaisseur-deployment-6db7c98c8d   3         3         0       3m17s   connaisseur   securesystemsengineering/connaisseur:v2.1.2   app.kubernetes.io/instance=connaisseur,app.kubernetes.io/nam
e=connaisseur,pod-template-hash=6db7c98c8d

Logs

[2021-09-16 05:06:31,182] INFO: 10.150.41.22 - - [16/Sep/2021 05:06:31] "GET /health HTTP/1.1" 200 -
[2021-09-16 05:06:35,148] INFO: 10.150.41.22 - - [16/Sep/2021 05:06:35] "GET /ready HTTP/1.1" 500 -
[2021-09-16 05:06:36,182] INFO: 10.150.41.22 - - [16/Sep/2021 05:06:36] "GET /health HTTP/1.1" 200 -
[2021-09-16 05:06:40,152] INFO: 10.150.41.22 - - [16/Sep/2021 05:06:40] "GET /ready HTTP/1.1" 500 -
[2021-09-16 05:06:41,182] INFO: 10.150.41.22 - - [16/Sep/2021 05:06:41] "GET /health HTTP/1.1" 200 -
[2021-09-16 05:06:45,148] INFO: 10.150.41.22 - - [16/Sep/2021 05:06:45] "GET /ready HTTP/1.1" 500 -
[2021-09-16 05:06:46,182] INFO: 10.150.41.22 - - [16/Sep/2021 05:06:46] "GET /health HTTP/1.1" 200 -
[2021-09-16 05:06:50,149] INFO: 10.150.41.22 - - [16/Sep/2021 05:06:50] "GET /ready HTTP/1.1" 500 -
[2021-09-16 05:06:51,182] INFO: 10.150.41.22 - - [16/Sep/2021 05:06:51] "GET /health HTTP/1.1" 200 -
[2021-09-16 05:06:55,149] INFO: 10.150.41.22 - - [16/Sep/2021 05:06:55] "GET /ready HTTP/1.1" 500 -

[xopham]

Hi @prasanjit-enginprogam 👋

That is strange. Do the pods ever become ready and then turn "non-ready"? What do you get when doing a kubectl describe on the pods?
Could you share your helm/values.yaml as well?

[xopham]

@gals-ma I did a test of my own:

1. setup an EKS Cluster according to: https://docs.aws.amazon.com/eks/latest/userguide/getting-started-console.html
2. added fargate nodes configs for namespaces default, connaisseur and kube-system and restarted the default addons (coredns, kubeproxy, vpc-cni) -> running system
3. setup a private repository on ECR according to: https://docs.aws.amazon.com/AmazonECR/latest/userguide/getting-started-console.html
4. to setup my local docker config, I followed step 1&2 of "Build, tag, and push a Docker image" of: https://docs.aws.amazon.com/AmazonECR/latest/userguide/getting-started-console.html
5. as a result my ~/.docker/config.json looks somewhat like the following:

{
	"auths": {
		"1234567890.dkr.ecr.us-west-2.amazonaws.com": {
			"auth": "<some-long-base64-string>"
		}
	}
}

5. I added that .docker/config.json as a kubernetes secret and configured connaisseur to use it for the respective validator according to: http://localhost:8000/validators/sigstore_cosign/#authentication .
6. I can check the secret via k get secrets -n connaisseur my-secret -o yaml
7. The respective validator and policy sections in my helm/values.yaml configuration look like:

(...)
validators:
(...)
- name: my-validator
  type: cosign
  auth:
    secret_name: my-secret
  trust_roots:
  - name: my-key
    key: |
      -----BEGIN PUBLIC KEY-----
      MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEef/kKgv5UpTbZO2hxMoUdrwdwIbo
      V2W5yzBlbxGLKjslcImDVxuOEEBdZNkJce10LD2hypqkL2VLiMP8a2heWA==
      -----END PUBLIC KEY-----
(...)
policy:
- pattern: "*:*"
  validator: my-validator
  with:
    trust_root: my-key
(...)

8. pushed some image to my ECR, e.g.: docker push 1234567890.dkr.ecr.us-west-2.amazonaws.com/my-ecr:demo
9. Running it unsigned was denied as expected: kubectl run demo --image=1234567890.dkr.ecr.us-west-2.amazonaws.com/my-ecr:demo
10. Signing it with cosign and running it again was admitted.

So that works nicely for me! In fact, I even tried without steps 5-7 which gives the same error as you had. So it is indeed authentication.

There is one caveat: Step 3 refers to AWS docs that already indicate that "This command provides an authorization token that is valid for 12 hours.". So, I suppose that when I try again tomorrow my previously created dockerconfigjson won't work anymore.
Will try to figure out tomorrow whether there is some sort of technical role or something available in AWS ECR to get a long-lived authorization token or something.

[gals-ma]

@xopham - This is very kind of you, highly appreciated.
I will walk through your instructions tomorrow and update.
About the last part, you might be able to set a cronjob that refreshes the token from AWS every X hours
See this article for reference - https://medium.com/@damitj07/how-to-configure-and-use-aws-ecr-with-kubernetes-rancher2-0-6144c626d42c
However, I would not suggest using the deployment from the article in terms of security, they are running a custom image that I did not verify in any way...
Thanks again for your amazing support.
Hope I will get Connaisseur working by tomorrow.

[gals-ma]

@xopham
Worked!!!

I followed your instructions and it worked perfectly.
All that is left now is to look for a solution for maintaining the secret in k8s.
Do you plan to address this in the code or should we aim for a workaround?

Thanks so much again!

[xopham]

@gals-ma happy to hear that it works now 🥳
I would recommend to aim for a workaround for now, as there seems no simply way to implement this in code.
When you use a solution as proposed above, make sure to restart the conny pods by restarting the rollout, as connaisseur mounts the credentials into the pod and will not automatically update without a restart

[shomeprasanjit]

Thanks, @xopham your approach worked for me. please go ahead and close this case.