Occasional failure calling the connaisseur webhook

[majorpeti007]

Hi, I need some help debugging an issue we hit when trying to add Connaisseur to our Azure Kubernetes Service (AKS) cluster.

We deploy our k8s resources using terraform, and we use the Connaisseur helm chart. We create about 50-60 k8s resources with a single terraform apply command. About 1 out of 10 times, the terraform apply fails with the following error:

Error: Internal error occurred: failed calling webhook "connaisseur-svc.connaisseur.svc": failed to call webhook: Post "[https://connaisseur-svc.connaisseur.svc:443/mutate?timeout=30s](https://connaisseur-svc.connaisseur.svc/mutate?timeout=30s)": read unix @->/tunnel-uds/proxysocket: read: connection reset by peer

We only saw this error when we ran 3 Connaisseur pods in the deployment. The Connaisseur logs don't show anything relevant, we already have the log level set to DEBUG.

[phbelitz]

We only saw this error when we ran 3 Connaisseur pods in the deployment.

Do you mean that the error does not occur when running only a single connaisseur pod? Or does scaling up the connaisseur pods help?

This looks similar to a problem we ourselves are stuggling with. For some reason the request do not come through to connaisseur at times. Now I can't say if Kubernetes fails to distribute the request or all Connaisseur pods are somehow busy.

I'd be interested how the error behaves when using e.g. 10 or more pods. Have you tried out somethigng like this?

[majorpeti007]

I meant that we had not seen the error with a single pod before but in an attempt to convince ourselves that using a single pod is temporarily fine for us, I reran terraform apply a couple of times and found the following:

• 1 Connaisseur pod - 20 terraform apply/destroy retries - 1 failure
• 3 Connaisseur pods - 20 terraform apply/destroy retries - 1 failure
• 10 Connaisseur pods - 40 terraform apply/destroy retries - 0 failure

[phbelitz]

Thank you for your efforts. Maybe changing the resource limits of connaisseur will give similar results?

Unfortunately I can't give you a proper solution to this, as we still don't know the exact issue causing this. What we plan on doing though is adding a caching function, so that connaisseur dosn't need to verify an image it already has verified shortly before. So if your 50-60 k8s resources share similar images, this might speed things up an may also solve this problem.

Anyways, thanks for your support again!