How to view / manage connections for fix max connections reached

[bediverus]

Hello,
sometimes I see a black screen with the message max connections reached: 15000 when opening the teleport web console, while tsh login also does not work.
I don't understand what "connections" means and how to manage them
I found the setting

connection_limits:
    max_connections: 15000

But this does not solve the problem, the teleport just takes longer, and then this error appears again
I'm testing Teleport and only three people are using it.
Сould you tell me how to troubleshoot this problem?

2023-11-15T07:32:45Z WARN [WEB]       Failed to write ping message error:[write tcp 127.0.0.1:3080->127.0.0.1:54492: write: broken pipe] web/conn_upgrade.go:125
bediverus@teleport-server:~$ date
Wed Nov 15 08:25:18 UTC 2023

I just tried to connect, but there is no entry in the logs

Teleport version: Teleport v14.1.1 git:api/v14.1.1-0-gfb6429e go1.21.3
teleport.yaml

auth_service:
  enabled: "yes"
  listen_addr: 127.0.0.1:3025
  cluster_name: "teleport.example.co"
  proxy_listener_mode: multiplex
  authentication:
    type: github
    second_factor: otp
    webauthn:
      rp_id: teleport.example.co

proxy_service:
  enabled: "yes"
  web_listen_addr: 127.0.0.1:3080
  public_addr: teleport.example.co:443
#  https_cert_file: /etc/letsencrypt/live/teleport.example.co/fullchain.pem
#  https_key_file: /etc/letsencrypt/live/teleport.example.co/privkey.pem
  https_keypairs: []
  https_keypairs_reload_interval: 0s
  acme: {}

deployed behind nginx as described in #26445

[webvictim]

Can you check how many active connections the Teleport process actually has? Something like sudo ss -tunap | grep teleport | wc -l on your Teleport server should do it.

It seems like nginx may not be closing the connections after use for some reason.

[webvictim]

Try setting this in your Teleport config file and restarting:

proxy_service:
  trust_x_forwarded_for: true

It's possible that Teleport is seeing all requests coming from the nginx server's IP rather than the client IP, which could lead to limits being hit. Setting this directive would cause it to get the client's IP from the X-Forwarded-For header sent by nginx instead, which should help with this.

(you may also need to configure nginx to send the header)

[bediverus]

Hello! Unfortunately, this did not help and the problem remained.
And I don't understand what "connections" are.
Is it possible to understand what it is and how connections are calculated?
Maybe I can look at these connections and somehow force the teleport to close them?

[bediverus]

Colleagues, topic up please
Has anyone encountered such a problem?
The only solution I came up with is to raise the port for monitoring
diag_addr: "127.0.0.1:3000"
and check/restart the service using cronjob
but this solution also doesn't work if the webserver is unavailable with the error "max connections reached"
curl http://127.0.0.1:3000/healthz returns {"status":"ok"}
I haven’t using "Teleport" for a long time, and this problem prevents me from switching to non-test mode :(

[webvictim]

I'm afraid I can't quite understand how this problem is only affecting you. The default limiter setting should allow up to 15000 connections every minute which is way more than enough for a small cluster.

Could you please share both your Teleport and nginx configs here with instructions to reproduce this issue?

[webvictim]

Question - are you using IPv6?

[webvictim]

Hi - this issue should have been fixed by #42799

This came out in versions 14.3.21, 15.4.3 and 16.0.0.

Please upgrade to a newer version and see if this helps!

[zzvara]

This is also an issue with public.ecr.aws/gravitational/teleport-distroless:16.4.2. Teleport is exposed through Traefik and Cloudflare.

[n0tl3an]

I confirm I am using version 16.4.2 on Debian 12, and I have the same issue, even though my report machine only shows 13 active connections. Please, do you have any idea how to fix this? I use Teleport as the main bastion, and it's a real problem not being able to use it for me.

teleport version
Teleport v16.4.2 git:v16.4.2-0-ge414b8f go1.22.7

sudo ss -tunap | grep teleport | wc -l
13

[webvictim]

Can you share more information about your environment, including your /etc/teleport.yaml file?

Do all your Teleport agents connect via reverse tunnel?

Do their connections all originate from the same IP address?

How many agents do you have connected?

After restarting the Teleport auth service, how long does it take before you see the "max connections reached" error?

[n0tl3an]

Hello, Thanks for your feedback, here is my complete Teleport configuration

version: v3
teleport:
  nodename: My_Node_Name
  data_dir: /var/lib/teleport
  log:
    output: stderr
    severity: INFO
    format:
      output: text
  ca_pin: ""
  diag_addr: ""

auth_service:
  enabled: "yes"
  listen_addr: 0.0.0.0:3025
  cluster_name: My_Cluster_Name
  proxy_listener_mode: multiplex

ssh_service:
  enabled: "yes"
  commands:
  - name: hostname
    command: [hostname]
    period: 1m0s

proxy_service:
  enabled: "yes"
  web_listen_addr: 0.0.0.0:3080
  public_addr: my_fqdn:443
  trust_x_forwarded_for: true

As for the connections, yes, they all go through an Nginx reverse proxy, and the connection IP is indeed always the same. Regarding the number of agents, I have 25.

Do you think the issue could be related to this kind of setup or my configuration? I don’t quite understand as I’ve been using Teleport for a long time and have never had this kind of issue until now. This is a new instance I provisioned after migrating services, and the old VM running Teleport under the same version never gave me this problem, even though the configuration, system, and network environments are identical, which leaves me a bit perplexed.

[webvictim]

Another question - are you or your agents using an IPv6 address to access the Teleport cluster? Does your nginx proxy host have an IPv6 address?

[n0tl3an]

No, I do not use IPv6. My VMs and networks are configured to use only IPv4, and this has always been the case without any problems. I'm sorry, I just noticed that I forgot to mention the time it takes for the problem to reappear after a Teleport restart; it's actually quite random, ranging between about 30 minutes and 3 hours.

[ghostklart]

+1 for the issue

it was working fine until i had to place WAF between Reverse Proxy (Caddy which forwards requests to teleport proxy container) and ISP. WAF used is SafeLine. After finding this topic tried the following:
added trust_x_forwarded_for: true to teleport proxy yaml config
and also just in case added header to Caddyfile directive header_up X-Forwarded-For "WAF IP address"

this issue still reproduces