DNS errors when accessing dynamically added app running in a kubernetes cluster

[alexkunin]

Hey guys,

I have a problem I'm hammering my head against for some time now. My goal is to be able to create my own app and db resources that refer to services running inside a Kubernetes cluster, without using NodePort service type and auto-discovery. But I can make it work, and getting DNS-related errors.

This is how it looks in the logs when I do something like tsh db connect <db-name>:

Jun 23 20:25:20 my-server teleport[136688]: 2024-06-23T20:25:20Z INFO [CA]        Generating TLS certificate <skip>
Jun 23 20:25:20 my-server teleport[136688]: 2024-06-23T20:25:20Z INFO [CA]        Generating TLS certificate <skip>
Jun 23 20:25:20 my-server teleport[136688]: 2024-06-23T20:25:20Z ERRO [DB:SERVIC] Failed to handle connection. addr:<skip>:57721 error:[
Jun 23 20:25:20 my-server teleport[136688]: ERROR REPORT:
Jun 23 20:25:20 my-server teleport[136688]: Original Error: *net.DNSError lookup <skip>.<skip>.svc.cluster.local: Temporary failure in name resolution
Jun 23 20:25:20 my-server teleport[136688]: Stack Trace:
Jun 23 20:25:20 my-server teleport[136688]:         github.com/gravitational/teleport/lib/srv/db/common/errors.go:161 github.com/gravitational/teleport/lib/srv/db/common.ConvertConnectError
Jun 23 20:25:20 my-server teleport[136688]:         <skip>
Jun 23 20:25:20 my-server teleport[136688]: User Message: lookup <skip>.<skip>.svc.cluster.local: Temporary failure in name resolution] db/server.go:959

Similar problem occurs when I open an application:

Jun 23 20:49:22 my-server teleport[136688]: 2024-06-23T20:49:22Z ERRO [APP:SERVI] Error forwarding to http://<skip>.<skip>.svc.cluster.local/, err: dial tcp: lookup <skip>.<skip>.svc.cluster.local: Temporary failure in name resolution reverseproxy/reverse_proxy.go:218

My setup looks like this:

1. Host 1:

• Teleport installed as a systemd unit directly on the host
• k0s controller
• Teleport Agent installed as a Helm chart

2. Host 2 and Host 3:

• k0s worker

I use Terraform to wind up some services (e.g. Postgres, Redis, code-server) and manually define Teleport resources. At different points in time services were running on k0s controller (which was in single mode at that time) and worker nodes.

If I define these resources with URI based on svc.cluster.local, I see DNS errors mentioned above.

If I allow agent to discover applications, I can see that generated Teleport resources contain svc.cluster.local URIs, and these auto-discovered applications work. I couldn't get DB auto-discovery working, so can't report on that.

I've switched agent image to -debug version and was able to confirm that cluster DNS resolution works fine there.

My guess is, auto-discovered apps are in some way "bound" to agent, and thus benefit from in-cluster DNS resolution, while my manually created apps are "bound" to Teleport cluster controller which uses host's (not Kubernetes') DNS resolution and thus fails to do proper look-ups.

Thanks!

[webvictim]

Could you please you share some example Teleport configs/values that would allow reproduction of your issue?

I think you are partly correct, but my understanding is that the app_service will always resolve DNS locally on the host where it's running. It'd be good to know the exact circumstances where lookups fail so we can investigate further.

[ak-purplesmart-ai]

I guess this is the answer I was looking for. Thank you for your help!

[ak-purplesmart-ai]

And, if somebody looks for the answer to similar question regarding DBs, the principle is the same -- add labels mentioned above to teleport_database resources, but also:

1. You need to enable agent's db_service by adding db role to agent's helm chart values along with this snippet:

databaseResources:
  - labels:
      '*': '*'

2. You need to generate a join token which for all roles mentioned in helm char values, e.g. standard Kubernetes cluster procedure does not include db role.

[ak-purplesmart-ai]

Aaand... it did not help. The issue comes and goes randomly, I have two service/pod pairs run next to each other (created by same module with different configuration which affects only names or resources), one of them is working fine, the other one causes DNS resolution errors in logs.

As far as I can tell, one is served by Teleport agent from inside of Kubernetes, the other one is served by main Teleport instlalation which is outside of Kubernetes.

@webvictim, can you unmark the selected answer, since it does not solve the issue, please?

[zmb3]

@webvictim's answer is correct.

You appear to be using dynamic registration. I recommend having a read through the docs to better understand how this feature works.

In a nutshell, you have registered an app named test with terraform that contains the following labels:

  metadata = {
    name = "test"
    labels = {
      "teleport.dev/kubernetes-cluster" = "cluster-1"
      "teleport.dev/origin" = "discovery-kubernetes"
    }
  }

Your app service is declaring that it can proxy the traffic for any and all apps with these labels:

app_service:
  enabled: true
  resources:
  - labels:
      teleport.dev/kubernetes-cluster: cluster-1
      teleport.dev/origin: discovery-kubernetes

If you are running multiple app_services with the same configuration (and it sounds like you are), then they will each handle traffic for the test app. If some of these app services are not able to resolve DNS for the app (because they're running outside the kube cluster where the kube service exists), then you would get exactly the behavior you describe.

To fix this, you need to make sure that only the app_services that can actually handle apps in kube cluster cluster-1 are declaring this label in their configuration.

[ak-purplesmart-ai]

Thank you, @zmb3, now it's clear what is wrong. The missing bit was: if several app services match app via labels, any of them can get the ownership of said app. For some reason (ahem, who said CSS...) I was sure that more specific selector wins.

Also, it seems application resources linger a bit in the system after deletion, so it wasn't always clear if I'm working with the latest configuration. Noticed this many times after doing tctl rm and then seeing some apps in Teleport UI and tsh app ls output.

Anyhow, I hope this explanation and solution will finally work, and won't need to get back to this discussion.

Thanks again!

P.S. I've seen that doc page during my installation and troubleshooting sessions, it seems all dots are there -- but not connections, and I couldn't connect the dots on my own.