Can't connect to DB with Teleport Consistently

[olawale-ib]

So my DBs have been set up and discovered by teleport config here
I have had mixed success connecting to the db by teleport (like 2 in 50+ attempts)

The iam role for teleport is here

Our dbs are created via terraform and doesn't have IAM authentication enabled

tsh login
tsh db login --db-name $db_name --db-user teleport $rds_name
tsh db connect $rds_name-rds-us-west-2-<account id>
OR
tsh proxy db --tunnel $rds_name-rds-us-west-2-<account id>

either way I get 👇🏽 this

psql: error: connection to server at "localhost" (::1), port 51295 failed: Connection refused
        Is the server running on that host and accepting TCP/IP connections?
connection to server at "localhost" (127.0.0.1), port 51295 failed: Could not connect to database:
  FATAL: password authentication failed for user "teleport" (SQLSTATE 28P01)
Make sure that Postgres user "teleport" has "rds_iam" role and Teleport database
agent's IAM policy has "rds-connect" permissions (note that IAM changes may
take a few minutes to propagate):
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "rds-db:connect",
            "Resource": "arn:aws:rds-db:us-west-2:<account id>:dbuser:db-HLCC23M6KIVMLPELNLGM2Q5FB4/*"
        }
    ]
}

ERROR: exit status 2

and yes rds-iam role is assigned to teleport, even the superadmin role is assigned to it too

I created a test db and enabled IAM authentication but got the same error

The 2 times I was able to connect, the logs and sessions showed up and were uploaded to s3 but didn't show up in the session recordings tab on the UI; teleport config is here but I use node-sync now

A question on the side, does teleport adds it own policy to the iam role? when creating my test db, this policy was removed since it wasn't in my code.

- inline_policy {
          - name   = "teleport.example.com-teleport-database-access" -> null
          - policy = jsonencode(
                {
                  - Statement = [
                      - {
                          - Action   = "rds-db:connect"
                          - Effect   = "Allow"
                          - Resource = [
                              - "arn:aws:rds-db:us-west-2:<account id>:dbuser:db-T7KJE7ZGXHG6SYTM3ZCTURNLGE/*",
                              - "arn:aws:rds-db:us-west-2:<account id>:dbuser:db-GH63IVMYWDCVNSSPEP7I6AID44/*",
                              - "arn:aws:rds-db:us-west-2:<account id>:dbuser:db-SWMQDRWB64LY33UHMTGKZH67II/*",
                              - "arn:aws:rds-db:us-west-2:<account id>:dbuser:db-MCJT4VTUEUNPDJUVSAX3WWD6GY/*",
                              - "arn:aws:rds-db:us-west-2:<account id>:dbuser:db-GBCO64A2XO7JVI5GLPQ3ZTJNAM/*",
                              - "arn:aws:rds-db:us-west-2:<account id>:dbuser:db-HLCC23M6KIVMLPELNLGM2Q5FB4/*",
                            ]
                        },
                    ]
                  - Version   = "2012-10-17"
                }
            ) -> null
        }

[webvictim]

I have a feeling IAM doesn't allow the use of wildcards in place of the cluster ID (the db-HAIIREIEEI... type strings you see in the policy that's being removed)

Teleport does create and manage its own role to specifically allow databases for this reason.

If you apply the policy using a specific cluster ID rather than a wildcard, wait a couple of minutes and then test that specific database via Teleport, does it work?

If not, what's in the logs on your Teleport db_service agent?

[olawale-ib]

I don't think it is the wildcard, this was the log I was getting previously

{"caller":"common/watcher.go:160","component":"watcher:cloud","error":"AccessDenied: User: arn:aws:sts::<teleportID>:assumed-role/teleport-role/1725575458981405569
is not authorized to perform: rds:DescribeDBClusters on resource: arn:aws:rds:us-west-2:<accountID>:cluster:* because no
identity-based policy allows the rds:DescribeDBClusters action\n\tstatus code: 403, request id: 35e5d57d-2e8a-412d-88cd-cf8e20cd4ea0",
"fetcher":{},"level":"warning","message":"Unable to fetch resources for db at AWS.","timestamp":"2024-09-05T22:30:59Z"}

So I added rds:DescribeDBClusters permission
Now I get this which shows that everything is perfect but I'm still getting psql: error: connection to server at "localhost" (::1), port 51295 failed: Connection refused error from before

{"caller":"reversetunnel/agent.go:575","component":"proxy:agent","leaseID":14,"level":"debug","message":"Transport request: teleport-transport.","target":"teleport.example.com:443","timestamp":"2024-09-05T23:58:04Z","trace.fields":
{"localCluster":"","targetCluster":"teleport.example.com"}}{"addr":{"IP":"X.X.X.X","Port":48050,"Zone":""},"caller":"db/server.go:933",
"component":"db:service","level":"debug","message":"Accepted connection.","timestamp":"2024-09-05T23:58:04Z"}
{"caller":"db/server.go:1139","component":"db:service","level":"debug","message":"Client identity: 
tlsca.Identity{Username:\"olawale\", Impersonator:\"\", Groups:[]string{\"access\", \"editor\", \"developer\"}, SystemRoles:[]string(nil), Usage:[]string{\"usage:db\"}, Principals:[]string{\"debian\", \"ubuntu\", \"olawale\", \"-teleport-internal-join\"}, 
KubernetesGroups:[]string{\"system:masters\"}, KubernetesUsers:[]string(nil), Expires:time.Date(2024, time.September, 6, 7, 
58, 4, 0, time.UTC), RouteToCluster:\"teleport.example.com\", KubernetesCluster:\"\", Traits:wrappers.Traits{\"github_teams\":
[]string{\"Eng\"}, \"kubernetes_groups\":[]string(nil), \"kubernetes_users\":[]string(nil), \"logins\":[]string{\"olawale\"}}, RouteToApp:tlsca.RouteToApp{SessionID:\"\", PublicAddr:\"\", ClusterName:\"\", Name:\"\", AWSRoleARN:\"\", 
AzureIdentity:\"\", GCPServiceAccount:\"\"}, TeleportCluster:\"teleport.example.com\", RouteToDatabase:tlsca.RouteToDatabase{ServiceName:\"service-rds-us-west-2-<accountID>\", Protocol:\"postgres\", 
Username:\"teleport\", Database:\"service\", Roles:[]string(nil)}, DatabaseNames:[]string{\"*\"}, DatabaseUsers:[]string{\"developer\", \"teleport\"}, MFAVerified:\"\", PreviousIdentityExpires:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), 
LoginIP:\"X.X.X.X\", PinnedIP:\"\", AWSRoleARNs:[]string(nil), AzureIdentities:[]string(nil), GCPServiceAccounts:[]string(nil), 
ActiveRequests:[]string(nil), DisallowReissue:false, Renewable:false, Generation:0x0, BotName:\"\", AllowedResourceIDs:[]types.ResourceID(nil), PrivateKeyPolicy:\"none\", ConnectionDiagnosticID:\"\", 
DeviceExtensions:tlsca.DeviceExtensions{DeviceID:\"\", AssetTag:\"\", CredentialID:\"\"}, UserType:\"\"}.","timestamp":"2024-
09-05T23:58:04Z"}{"caller":"db/server.go:1152","component":"db:service","level":"debug","message":"Will connect to 
database \"service-rds-us-west-2-<accountID>\" at service.c3c4slz0qzly.us-west-2.rds.amazonaws.com:5432.","timestamp":"2024-09-05T23:58:04Z"}
{"caller":"db/server.go:1175","component":"db:service","level":"debug","message":"Session context: db[service-rds-us-west-2-<accountID>] identity[olawale] dbUser[teleport] dbName[service] autoCreate[DB_USER_MODE_UNSPECIFIED] dbRoles[].","timestamp":"2024-09-05T23:58:04Z"}
{"caller":"db/server.go:1229","component":"db:service","level":"debug","message":"Creating tracker for session 4e881950-7065-4fdb-bc9c-0189e8b99c2b","timestamp":"2024-09-05T23:58:04Z"}
{"caller":"postgres/engine.go:206","component":"db:service","db":"service-rds-us-west-2-<accountID>","id":"4e881950-7065-4fdb-bc9c-0189e8b99c2b","level":"debug","message":"Received startup message: 
\u0026pgproto3.StartupMessage{ProtocolVersion:0x30000, Parameters:map[string]string{\"application_name\":\"psql\", \"client_encoding\":\"SQL_ASCII\", \"database\":\"service\", \"user\":\"teleport\"}}.","timestamp":"2024-09-05T23:58:05Z"}
{"caller":"db/server.go:1021","component":"db:service","level":"debug","message":"Real client IP X.X.X.X","timestamp":"2024-09-05T23:58:05Z"}
{"caller":"cloud/clients.go:807","component":null,"level":"debug","message":"Initializing AWS session for region us-west-2 using environment credentials.","timestamp":"2024-09-05T23:58:05Z"}{"caller":"common/auth.go:193","component":"db:auth","level":"debug","message":
"Generating RDS auth token for db[service-rds-us-west-2-<accountID>] identity[olawale] dbUser[teleport] dbName[service] 
autoCreate[DB_USER_MODE_UNSPECIFIED] dbRoles[].","timestamp":"2024-09-05T23:58:05Z"}
{"caller":"reversetunnel/agent.go:652","component":"proxy:agent","leaseID":14,"level":"debug","message":"Received discovery request: [a8674e00-9987-4a43-81b0-1ceb89d4486b]","target":"teleport.example.com:443","timestamp":"2024-
09-05T23:58:16Z","trace.fields":{"localCluster":"","targetCluster":"teleport.example.com"}}

[olawale-ib]

Found the issue. Previous engineer created the teleport user on each instance with the required role but it seems it was removed. Created the user with the required permission and it worked.