Getting Teleport timeout on healthy servers

[nsantiago2719]

Hi,

I just want to ask if anyone have encountered this before:

Jun 23 03:45:28 ip-172-11-12-237 teleport[453]: ERRO             read tcp 172.11.12.237:3024->172.11.2.152:56062: read: connection reset by peer sshutils/server.go:667
Jun 23 03:47:15 ip-172-11-12-237 teleport[453]: ERRO             read tcp 172.11.12.237:3023->172.11.2.152:5129: read: connection reset by peer sshutils/server.go:667
Jun 23 03:48:47 ip-172-11-12-237 teleport[453]: ERRO             read tcp 172.11.12.237:3023->172.11.2.152:20770: read: connection reset by peer sshutils/server.go:667
Jun 23 03:48:58 ip-172-11-12-237 teleport[453]: ERRO [PROXY:SER] "proxy2021/06/23 03:48:58 http: TLS handshake error from 103.130.177.32:33287: EOF\n" utils/cli.go:304
Jun 23 03:48:58 ip-172-11-12-237 teleport[453]: ERRO [PROXY:SER] "proxy2021/06/23 03:48:58 http: TLS handshake error from 103.130.177.32:36733: EOF\n" utils/cli.go:304
Jun 23 03:49:24 ip-172-11-12-237 teleport[453]: ERRO [DYNAMODB]  "Poll streams returned with error: RequestError: send request failed\ncaused by: Post \"https://streams.dynamodb.ap-southeast-1.amazonaws.com/\": read tcp 172.11.12.237:4347>
Jun 23 03:49:24 ip-172-11-12-237 teleport[453]: ERRO [DYNAMODB]  "Poll streams returned with error: RequestError: send request failed\ncaused by: Post \"https://dynamodb.ap-southeast-1.amazonaws.com/\": read tcp 172.11.12.237:50432->52.94>
Jun 23 03:49:25 ip-172-11-12-237 teleport[453]: ERRO [DYNAMODB]  "Poll streams returned with error: RequestError: send request failed\ncaused by: Post \"https://streams.dynamodb.ap-southeast-1.amazonaws.com/\": read tcp 172.11.12.237:4345>
Jun 23 03:49:45 ip-172-11-12-237 teleport[453]: ERRO             read tcp 172.11.12.237:3023->172.11.2.152:8709: read: connection reset by peer sshutils/server.go:667
Jun 23 03:52:05 ip-172-11-12-237 teleport[453]: ERRO             read tcp 172.11.12.237:3023->172.11.2.152:29466: read: connection reset by peer sshutils/server.go:667

So I was able to create a cluster and added some nodes into it but I noticed some problems while using the CLI and the Dashboard. Sometimes I'm having a timeout in CLI while the website is working well but there are instances that the browser is not working but I can still use the CLI. To add some more information I have 2 instances running proxy and auth, they are load balance using network load balancer.

I'm using Teleport v6.2 by the way.

[webvictim]

It looks a lot like your connectivity on that server isn't so good - the connections to DynamoDB being reset and the timeouts/disconnections overall seem like there's an intermittent connectivity problem. I would investigate the server's networking as the first port of call.

[nsantiago2719]

I'll get back to you on this. Seems we need to recheck the server type used too since all servers have the same issue.

[nsantiago2719]

So I've tried rebuilding the whole cluster in a new VPC and Instance(also upgrade some) and still experiencing the same error.