Increase startup and liveness probe timeout for linkerd-destination

[kristjankullerkann-wearemp]

What problem are you trying to solve?

We are seeing startup and liveness probe warnings for linkerd-destination from time to time:

Liveness probe failed: Get "http://10.0.0.7:9990/live": context deadline exceeded (Client.Timeout exceeded while awaiting headers) | linkerd-destination-744c8f574c-vx95n
Liveness probe failed: Get "http://10.0.0.54:9997/ping": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

The issue and solution is well described in:
#8235

How should the problem be solved?

I am suggesting to make this configurable in helm chart and default to current values.

destinationProbes:
  livenessProbe:
    timeoutSeconds: 1
  readinessProbe:
    timeoutSeconds: 1

Any alternatives you've considered?

Raise timeouts in https://github.com/linkerd/linkerd2/blob/main/charts/linkerd-control-plane/templates/destination.yaml

How would users interact with this feature?

No response

Would you like to work on this feature?

yes

[alpeb]

Can you share what linkerd version you're on? There was an issue with readiness probes taking a long time to succeed, but that was fixed some time ago.

[kristjankullerkann-wearemp]

@alpeb sorry for the late reply but I am on edge-24.7.5

I have increased timeout from 1 to 5 seconds and the the probes do not fail anymore.

[alpeb]

Hi @kristjankullerkann-wearemp, I think it would be valuable to add configurability to the probes. Are you up to push a PR with this? I'd be glad to provide some guidance. It'd be great to implement this generically, both for liveness and readiness probes, with a default that could be overridden by each of the control plane containers.

[kristjankullerkann-wearemp]

@alpeb sure I will create a PR.