Teleport on kubernetes: proxy pod hangs in Init:0/1 status #29180

urbaman · 2023-07-16T16:45:55Z

urbaman
Jul 16, 2023

Hi,

Trying to install teleport on baremetal kubernetes, in HA mode.

The proxy pod hangs in Init:0/1 status, and logs say

Defaulted container "teleport" out of: teleport, wait-auth-update (init)
Error from server (BadRequest): container "teleport" in pod "teleport-cluster-proxy-77d64d848-8dhcc" is waiting to start: PodInitializing```

Is there a way to solve/debug it?

Thank you.

urbaman · 2023-07-16T16:58:55Z

urbaman
Jul 16, 2023
Author

Also in sigle replica mode I have the same result

0 replies

urbaman · 2023-07-17T09:39:04Z

urbaman
Jul 17, 2023
Author

8 replies

urbaman Jul 20, 2023
Author

Also for debugging, I have all of the auth pods running, and the init container from the proxy pod seems not to be able to see the auth pods:

NAME                                      READY   STATUS     RESTARTS        AGE   IP             NODE    NOMINATED NODE   READINESS GATES
teleport-cluster-auth-cf847cf44-grrx7     2/2     Running    0               89m   10.10.14.61    k8w1    <none>           <none>
teleport-cluster-proxy-7dddd98984-2j6jj   0/1     Init:0/1   8 (9m27s ago)   89m   10.10.157.49   k8cp1   <none>           <none>
teleport-cluster-proxy-7dddd98984-s2cmq   0/1     Init:0/1   8 (9m30s ago)   89m   10.10.14.54    k8w1    <none>           <none>

  wait-auth-update:
    Container ID:  containerd://2adfe972d1907eeba8cc7f3c32403243c2a5a39f29ebc2728e92f514c6c19851
    Image:         public.ecr.aws/gravitational/teleport-distroless:13.2.2
    Image ID:      public.ecr.aws/gravitational/teleport-distroless@sha256:51da5d14aa5a61ce1c6a7b2111d8b7fcf8bb579ab161ffccd05eab19ca1bf1cb
    Port:          <none>
    Host Port:     <none>
    Command:
      teleport
      wait
      no-resolve
      teleport-cluster-auth-v12.teleport-cluster.svc.cluster.local
    State:          Running
      Started:      Mon, 17 Jul 2023 11:29:24 +0200
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Mon, 17 Jul 2023 11:19:23 +0200
      Finished:     Mon, 17 Jul 2023 11:29:23 +0200
    Ready:          False
    Restart Count:  8
    Environment:    <none>
    Mounts:         <none>

urbaman Jul 20, 2023
Author

Hi,

Here are some more logs and insights with a test deployment of Teleport (now 13.2.3), showing the v12 and v13 without CLusterIPs or Ports, so not being reachable as show form the other logs.

Teleport deployed following point by point the "Deploy Teleport on Kubernetes" guide, just selecting different namespaces and deployment name to differentiate from the prod deployment.

Hope it helps.

ubuntu@k8cp1:~$ kubectl get all -n teleport-cluster-test
NAME                                               READY   STATUS     RESTARTS   AGE
pod/teleport-cluster-test-auth-995b64fcf-gd465     1/1     Running    0          34s
pod/teleport-cluster-test-proxy-774644bfc5-65lm6   0/1     Init:0/1   0          34s

NAME                                     TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
service/teleport-cluster-test            LoadBalancer   172.25.199.113   10.0.50.102   443:31332/TCP       36s
service/teleport-cluster-test-auth       ClusterIP      172.21.95.104    <none>        3025/TCP,3026/TCP   36s
service/teleport-cluster-test-auth-v12   ClusterIP      None             <none>        <none>              36s
service/teleport-cluster-test-auth-v13   ClusterIP      None             <none>        <none>              36s

NAME                                          READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/teleport-cluster-test-auth    1/1     1            1           35s
deployment.apps/teleport-cluster-test-proxy   0/1     1            0           35s

NAME                                                     DESIRED   CURRENT   READY   AGE
replicaset.apps/teleport-cluster-test-auth-995b64fcf     1         1         1       34s
replicaset.apps/teleport-cluster-test-proxy-774644bfc5   1         1         0       34s



ubuntu@k8cp1:~$ kubectl logs -n teleport-cluster-test teleport-cluster-test-proxy-774644bfc5-65lm6
Defaulted container "teleport" out of: teleport, wait-auth-update (init)
Error from server (BadRequest): container "teleport" in pod "teleport-cluster-test-proxy-774644bfc5-65lm6" is waiting to start: PodInitializing


ubuntu@k8cp1:~$ kubectl logs -n teleport-cluster-test teleport-cluster-test-proxy-774644bfc5-65lm6 -c wait-auth-update
2023-07-20T18:42:48Z ERRO             error when resolving domain teleport-cluster-test-auth-v12.teleport-cluster-test.svc.cluster.local : lookup teleport-cluster-test-auth-v12.teleport-cluster-test.svc.cluster.local: No address associated with hostname common/wait.go:151
2023-07-20T18:42:56Z ERRO             error when resolving domain teleport-cluster-test-auth-v12.teleport-cluster-test.svc.cluster.local : lookup teleport-cluster-test-auth-v12.teleport-cluster-test.svc.cluster.local: No address associated with hostname common/wait.go:151


ubuntu@k8cp1:~$ kubectl exec -it samplepod -- ping teleport-cluster-test-auth-v12.teleport-cluster-test.svc.cluster.local
ping: teleport-cluster-test-auth-v12.teleport-cluster-test.svc.cluster.local: Name or service not known
command terminated with exit code 2

hugoShaka Jul 25, 2023
Maintainer

Thank you for the logs, this is really useful.

I tried to reproduce but was unable to get this exact DNS error. I suspect this is caused by a weird DNS response or a client which cannot perform DNS requests at all. Could you give more details about the Kubernetes cluster you are running in?

How it was deployed: GKE, EKS, Rancher, IBM, AKS, kubespray, ...
Which Kubernets version
Are you running a custom image?
Which CPU architecture: arm64 or amd64
What in-cluster DNS are you using: coredns, kubedns (the old skydns-based one), cloudDNS by GCP, ...
Any security hardening that could block teleport from resolving DNS entries: custom seccomp profile, hardened AppArmor, ...
Any other DNS-related infrastructure that could be relevant? (node-local DNS caches for example)

urbaman Jul 25, 2023
Author

Deployed with kubeadm on Ubuntu 22.04
Version 1.27.2
?
Amd64
coredns
no hardening
no relevant DNS infrastructure

Using Calico and Multus with Wereabouts for using a second network on some pods (Longhorn Storage).

The strange thing is that the two v12 and v13 services are not enabled (without an IP), so they're not reachable I think:

NAME                                     TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
service/teleport-cluster-test            LoadBalancer   172.25.199.113   10.0.50.102   443:31332/TCP       36s
service/teleport-cluster-test-auth       ClusterIP      172.21.95.104    <none>        3025/TCP,3026/TCP   36s
service/teleport-cluster-test-auth-v12   ClusterIP      None             <none>        <none>              36s
service/teleport-cluster-test-auth-v13   ClusterIP      None             <none>        <none>              36s

Teleport seems to be reaching everything else, also the "normal" teleport-cluster-test-auth

hugoShaka Jul 25, 2023
Maintainer

Thank you for the information, I will try to reproduce when I have the time.

The 12 and v13 don't have an IP because they are headless. They are not meant for loadbalancing but pod discovery. Resolving them returns the list of individual pods, and not a LB IP. For v12, this list is empty, for v13 it contains 1 entry per auth pod.

Something wrong is happening with the empty response, this is not a regular NXDOMAIN error for some reason. Without reproducing, I can't say if this error can be safely considered a success and let the initContainer exit or retry.

urbaman · 2023-07-19T07:50:19Z

urbaman
Jul 19, 2023
Author

Ok, solved.

I had to manually edit the proxy deployment to remove the initcontainer, that searched for the wrong service (v12).

I think that should be fixed or more detailed/manageable

1 reply

hugoShaka Jul 20, 2023
Maintainer

The proxy searching for the v12 service is intentional and required. It ensures no v12 auth is still running during a v12 to v13 update. What was the error log in this container?

urbaman · 2023-07-20T19:12:45Z

urbaman
Jul 20, 2023
Author

Sorry for messing around with the comments...

0 replies

sms-astanley · 2023-09-22T01:06:58Z

sms-astanley
Sep 22, 2023

I seem to have hit a similar issue.

> kubectl get pods
NAME                              READY   STATUS     RESTARTS       AGE
teleport-auth-5b4f5bc87f-k5kcg    1/1     Running    0              11m
teleport-proxy-5687844c7f-59h56   0/1     Init:0/1   1 (118s ago)   11m
teleport-proxy-5687844c7f-zgbqp   0/1     Init:0/1   1 (118s ago)   11m

> kubectl logs -f teleport-proxy-5687844c7f-59h56 -c wait-auth-update
...
2023-09-22T01:04:16Z WARN             temporary error when resolving domain teleport-auth-v13.teleport.svc.cluster.local : lookup teleport-auth-v13.teleport.svc.cluster.local: Temporary failure in name resolution common/wait.go:144
2023-09-22T01:04:25Z WARN             temporary error when resolving domain teleport-auth-v13.teleport.svc.cluster.local : lookup teleport-auth-v13.teleport.svc.cluster.local: Temporary failure in name resolution common/wait.go:144
2023-09-22T01:04:35Z WARN             temporary error when resolving domain teleport-auth-v13.teleport.svc.cluster.local : lookup teleport-auth-v13.teleport.svc.cluster.local: Temporary failure in name resolution common/wait.go:144
2023-09-22T01:04:45Z WARN             temporary error when resolving domain teleport-auth-v13.teleport.svc.cluster.local : lookup teleport-auth-v13.teleport.svc.cluster.local: Temporary failure in name resolution common/wait.go:144
2023-09-22T01:04:54Z WARN             temporary error when resolving domain teleport-auth-v13.teleport.svc.cluster.local : lookup teleport-auth-v13.teleport.svc.cluster.local: Temporary failure in name resolution common/wait.go:144
2023-09-22T01:05:03Z WARN             temporary error when resolving domain teleport-auth-v13.teleport.svc.cluster.local : lookup teleport-auth-v13.teleport.svc.cluster.local: Temporary failure in name resolution common/wait.go:144
2023-09-22T01:05:13Z WARN             temporary error when resolving domain teleport-auth-v13.teleport.svc.cluster.local : lookup teleport-auth-v13.teleport.svc.cluster.local: Temporary failure in name resolution common/wait.go:144

2023-09-22T01:05:19.431625165Z ERROR REPORT:
Original Error: *trace.LimitExceededError timeout (10m0s) reached, but domain &#39;teleport-auth-v13.teleport.svc.cluster.local&#39; is still resolving
2023-09-22T01:05:19.431633882Z Stack Trace:
2023-09-22T01:05:19.431637454Z 	github.com/gravitational/teleport/tool/teleport/common/wait.go:115 github.com/gravitational/teleport/tool/teleport/common.waitNoResolve
2023-09-22T01:05:19.431640968Z 	github.com/gravitational/teleport/tool/teleport/common/wait.go:61 github.com/gravitational/teleport/tool/teleport/common.onWaitNoResolve
2023-09-22T01:05:19.431643982Z 	github.com/gravitational/teleport/tool/teleport/common/teleport.go:531 github.com/gravitational/teleport/tool/teleport/common.Run
2023-09-22T01:05:19.431647859Z 	github.com/gravitational/teleport/tool/teleport/main.go:31 main.main
2023-09-22T01:05:19.431650831Z 	runtime/proc.go:267 runtime.main
2023-09-22T01:05:19.431653434Z 	runtime/asm_amd64.s:1650 runtime.goexit
2023-09-22T01:05:19.431656711Z User Message: timeout (10m0s) reached, but domain &#39;teleport-auth-v13.teleport.svc.cluster.local&#39; is still resolving

2 replies

sms-astanley Sep 22, 2023

This recommendation allowed me to move forward - #22285

Btijmen Jan 25, 2024

Same here, running v14.3.3, and the linked suggestion skipping the init container altogether also allowed me to successfully deploy

edit a few days later:
I left this in a non-working state on Friday. When I returned to work on Monday, the init container had successfully completed and the proxy container was running perfectly. I have no idea why "wait a few days" solved the issue, but I haven't even been able to replicate the issue since.
(that's a problem in itself, because I can't roll this out to production without understanding why this issue occurs and how to prevent it)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Teleport on kubernetes: proxy pod hangs in Init:0/1 status #29180

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 5 comments 11 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

Teleport on kubernetes: proxy pod hangs in Init:0/1 status #29180

urbaman Jul 16, 2023

Replies: 5 comments · 11 replies

urbaman Jul 16, 2023 Author

urbaman Jul 17, 2023 Author

urbaman Jul 20, 2023 Author

urbaman Jul 20, 2023 Author

hugoShaka Jul 25, 2023 Maintainer

urbaman Jul 25, 2023 Author

hugoShaka Jul 25, 2023 Maintainer

urbaman Jul 19, 2023 Author

hugoShaka Jul 20, 2023 Maintainer

urbaman Jul 20, 2023 Author

sms-astanley Sep 22, 2023

sms-astanley Sep 22, 2023

Btijmen Jan 25, 2024

urbaman
Jul 16, 2023

Replies: 5 comments 11 replies

urbaman
Jul 16, 2023
Author

urbaman
Jul 17, 2023
Author

urbaman Jul 20, 2023
Author

urbaman Jul 20, 2023
Author

hugoShaka Jul 25, 2023
Maintainer

urbaman Jul 25, 2023
Author

hugoShaka Jul 25, 2023
Maintainer

urbaman
Jul 19, 2023
Author

hugoShaka Jul 20, 2023
Maintainer

urbaman
Jul 20, 2023
Author

sms-astanley
Sep 22, 2023