From b2b349c30ff531b48133d180bfc989e7dd6d80a6 Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Wed, 14 Feb 2024 14:41:18 +0000 Subject: [PATCH 1/9] clarify ssh tunnel dbt connection error this pr clarifies the ssh tunnel connection error. previously it was wordy and hard to read. this was per feedback from a user: https://dbt-labs.slack.com/archives/C067EUS2D7S/p1704912493484739 --- .../connect-redshift-postgresql-alloydb.md | 59 +++++++++---------- 1 file changed, 28 insertions(+), 31 deletions(-) diff --git a/website/docs/docs/cloud/connect-data-platform/connect-redshift-postgresql-alloydb.md b/website/docs/docs/cloud/connect-data-platform/connect-redshift-postgresql-alloydb.md index 03303ea8d52..b6d3ea26498 100644 --- a/website/docs/docs/cloud/connect-data-platform/connect-redshift-postgresql-alloydb.md +++ b/website/docs/docs/cloud/connect-data-platform/connect-redshift-postgresql-alloydb.md @@ -63,39 +63,36 @@ chmod 600 ~/.ssh/authorized_keys The Bastion server should now be ready for dbt Cloud to use as a tunnel into the Redshift environment. -#### Intermittent connection issues - -
- Database Error - could not connect to server: Connection timed out - You will have the following components when you configure a connection to a database using an SSH tunnel:
- - An Elastic Load Balancer (ELB) or Network Load Balancing (NLB) instance.
- - A bastion host (or jump server) running the sshd process.
- - A Database (such as Redshift cluster)

- -dbt Cloud establishes an SSH tunnel connection through the ELB or NLB to the sshd process. This is responsible for routing traffic to the database. When dbt initiates a job run, an SSH tunnel is created at the start of the run. If this SSH tunnel fails at any point, the job will also fail.
- - The most common causes of tunnel failures are:
- - The SSH daemon terminates the session due to an idle timeout.
- - The ELB or NLB terminates the connection when it's idle.
- -dbt Cloud sets a value for its SSH tunnel called `ServerAliveInterval` and `ServerAliveCountMax` that polls the connection every 30 seconds and the underlying OS in our run "pods" will terminate the connection if the `sshd` process fails to respond after 300s. This will, in many cases, prevent an idle timeout entirely so long as the customer is not using ELB with a firewall-level idle timeout of less than 30 seconds. However, if the customer is using ELB and is using an Idle Connection Timeout of less than 30s, this will be insufficient to prevent tunnels from being terminated.
- -Some versions of Linux used on bastion hosts use a version of `sshd` with additional idle timeout settings `ClientAliveCountMax`. This value sets the number of client alive messages that may be sent without `sshd` receiving any messages back from the client. If this threshold is reached while client alive messages are being sent, `sshd` will disconnect the client, terminating the session. The client-alive mechanism is helpful when the client or server needs to know when a connection has become inactive. The default value is 3.

- -`ClientAliveInterval`:
-This value sets a timeout interval in seconds after which if no data has been received from the client, `sshd` will send a message through the encrypted channel to request a response from the client. The default is 0, indicating that these messages will not be sent to the client.
- -Using default values, tunnels could be terminated prematurely by `sshd`. To solve this problem, the `/etc/ssh/sshd_config` file on the bastion host can be configured with the following values:

-- `ClientAliveCountMax` 10
-- `ClientAliveInterval` 30
-where `ClientAliveCountMax` should be set to a non-zero value and `ClientAliveInterval` should be a value less than the ELB or NLB idle timeout value.
- -With these settings, unresponsive SSH clients will be disconnected after approximately 300 seconds, helping to prevent tunnel failures. -
- ## Configuration -To learn how to optimize performance with data platform-specific configurations in dbt Cloud, refer to [Redshift-specific configuration](/reference/resource-configs/redshift-configs). +To optimize performance with data platform-specific configurations in dbt Cloud, refer to [Redshift-specific configuration](/reference/resource-configs/redshift-configs). To grant users or roles database permissions (access rights and privileges), refer to the [Redshift permissions](/reference/database-permissions/redshift-permissions) page or [Postgres permissions](/reference/database-permissions/postgres-permissions) page. + +## FAQs + + +When setting up a database connection using an SSH tunnel, you typically need these parts: + +- A Load Balancer (like ELB or NLB) to manage traffic. +- A bastion host (or jump server) that runs the SSH protocol, acting as a secure entry point. +- The database itself (such as a Redshift cluster). + +dbt Cloud uses an SSH tunnel to connect through the Load Balancer to the database. This tunnel is set up at the start of any dbt job run. If the tunnel drops, the job fails. + +Tunnel failures usually happen because: + +- The SSH service stops if there's no activity for too long. +- The Load Balancer cuts off the connection if it's idle. +- dbt Cloud tries to keep the connection alive by checking in every 30 seconds, and the system will end the connection if there's no response from the SSH service after 300 seconds. This helps avoid drops due to inactivity unless the Load Balancer's timeout is less than 30 seconds. + +Bastion hosts might have additional SSH settings to disconnect inactive clients after several checks without a response. By default, it checks three times. + +To prevent premature disconnections, you can adjust the settings on the bastion host: + +- `ClientAliveCountMax 10` — Allows more checks before deciding the client is inactive. +- `ClientAliveInterval 30` — Checks for client activity every 30 seconds. +These adjustments ensure that inactive SSH clients are disconnected after about 300 seconds, reducing the chance of tunnel failures. + + From 7e5d033d6bbfd75e633ebfea4db3be5d68aea41c Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Fri, 1 Mar 2024 12:00:16 +0000 Subject: [PATCH 2/9] Update website/docs/docs/cloud/connect-data-platform/connect-redshift-postgresql-alloydb.md Co-authored-by: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> --- .../connect-redshift-postgresql-alloydb.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/cloud/connect-data-platform/connect-redshift-postgresql-alloydb.md b/website/docs/docs/cloud/connect-data-platform/connect-redshift-postgresql-alloydb.md index b6d3ea26498..f60b589bc34 100644 --- a/website/docs/docs/cloud/connect-data-platform/connect-redshift-postgresql-alloydb.md +++ b/website/docs/docs/cloud/connect-data-platform/connect-redshift-postgresql-alloydb.md @@ -73,7 +73,7 @@ To grant users or roles database permissions (access rights and privileges), ref ## FAQs -When setting up a database connection using an SSH tunnel, you typically need these parts: +When setting up a database connection using an SSH tunnel, you need the following components: - A Load Balancer (like ELB or NLB) to manage traffic. - A bastion host (or jump server) that runs the SSH protocol, acting as a secure entry point. From 01d7df041f11104c684d97be91c81b6753f9d567 Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Fri, 1 Mar 2024 12:00:37 +0000 Subject: [PATCH 3/9] Update website/docs/docs/cloud/connect-data-platform/connect-redshift-postgresql-alloydb.md Co-authored-by: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> --- .../connect-redshift-postgresql-alloydb.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/cloud/connect-data-platform/connect-redshift-postgresql-alloydb.md b/website/docs/docs/cloud/connect-data-platform/connect-redshift-postgresql-alloydb.md index f60b589bc34..65da4c2b2bc 100644 --- a/website/docs/docs/cloud/connect-data-platform/connect-redshift-postgresql-alloydb.md +++ b/website/docs/docs/cloud/connect-data-platform/connect-redshift-postgresql-alloydb.md @@ -75,7 +75,7 @@ To grant users or roles database permissions (access rights and privileges), ref When setting up a database connection using an SSH tunnel, you need the following components: -- A Load Balancer (like ELB or NLB) to manage traffic. +- A load balancer (like ELB or NLB) to manage traffic. - A bastion host (or jump server) that runs the SSH protocol, acting as a secure entry point. - The database itself (such as a Redshift cluster). From 4b0be4981f1e89b0d23809a092f1f17f39ffa6c8 Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Fri, 1 Mar 2024 12:00:52 +0000 Subject: [PATCH 4/9] Update website/docs/docs/cloud/connect-data-platform/connect-redshift-postgresql-alloydb.md Co-authored-by: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> --- .../connect-redshift-postgresql-alloydb.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/cloud/connect-data-platform/connect-redshift-postgresql-alloydb.md b/website/docs/docs/cloud/connect-data-platform/connect-redshift-postgresql-alloydb.md index 65da4c2b2bc..958f3969678 100644 --- a/website/docs/docs/cloud/connect-data-platform/connect-redshift-postgresql-alloydb.md +++ b/website/docs/docs/cloud/connect-data-platform/connect-redshift-postgresql-alloydb.md @@ -79,7 +79,7 @@ When setting up a database connection using an SSH tunnel, you need the followin - A bastion host (or jump server) that runs the SSH protocol, acting as a secure entry point. - The database itself (such as a Redshift cluster). -dbt Cloud uses an SSH tunnel to connect through the Load Balancer to the database. This tunnel is set up at the start of any dbt job run. If the tunnel drops, the job fails. +dbt Cloud uses an SSH tunnel to connect through the load balancer to the database. This connection is established at the start of any dbt job run. If the tunnel connection drops, the job fails. Tunnel failures usually happen because: From f794f0bc3ca29cf93b97df9ecfb6800e2bd8ff4d Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Fri, 1 Mar 2024 12:01:08 +0000 Subject: [PATCH 5/9] Update website/docs/docs/cloud/connect-data-platform/connect-redshift-postgresql-alloydb.md Co-authored-by: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> --- .../connect-redshift-postgresql-alloydb.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/cloud/connect-data-platform/connect-redshift-postgresql-alloydb.md b/website/docs/docs/cloud/connect-data-platform/connect-redshift-postgresql-alloydb.md index 958f3969678..36f93c6214d 100644 --- a/website/docs/docs/cloud/connect-data-platform/connect-redshift-postgresql-alloydb.md +++ b/website/docs/docs/cloud/connect-data-platform/connect-redshift-postgresql-alloydb.md @@ -83,7 +83,7 @@ dbt Cloud uses an SSH tunnel to connect through the load balancer to the databas Tunnel failures usually happen because: -- The SSH service stops if there's no activity for too long. +- The SSH daemon times out if it's idle for too long. - The Load Balancer cuts off the connection if it's idle. - dbt Cloud tries to keep the connection alive by checking in every 30 seconds, and the system will end the connection if there's no response from the SSH service after 300 seconds. This helps avoid drops due to inactivity unless the Load Balancer's timeout is less than 30 seconds. From ea71e1a69bf9c52c60c7b866721aa16f69de2da0 Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Fri, 1 Mar 2024 12:01:14 +0000 Subject: [PATCH 6/9] Update website/docs/docs/cloud/connect-data-platform/connect-redshift-postgresql-alloydb.md Co-authored-by: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> --- .../connect-redshift-postgresql-alloydb.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/cloud/connect-data-platform/connect-redshift-postgresql-alloydb.md b/website/docs/docs/cloud/connect-data-platform/connect-redshift-postgresql-alloydb.md index 36f93c6214d..d811457cad4 100644 --- a/website/docs/docs/cloud/connect-data-platform/connect-redshift-postgresql-alloydb.md +++ b/website/docs/docs/cloud/connect-data-platform/connect-redshift-postgresql-alloydb.md @@ -84,7 +84,7 @@ dbt Cloud uses an SSH tunnel to connect through the load balancer to the databas Tunnel failures usually happen because: - The SSH daemon times out if it's idle for too long. -- The Load Balancer cuts off the connection if it's idle. +- The load balancer cuts off the connection if it's idle. - dbt Cloud tries to keep the connection alive by checking in every 30 seconds, and the system will end the connection if there's no response from the SSH service after 300 seconds. This helps avoid drops due to inactivity unless the Load Balancer's timeout is less than 30 seconds. Bastion hosts might have additional SSH settings to disconnect inactive clients after several checks without a response. By default, it checks three times. From 7f8aa0991728fcc3e12a1ce1d0b3c8b2cc203792 Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Fri, 1 Mar 2024 12:01:37 +0000 Subject: [PATCH 7/9] Update website/docs/docs/cloud/connect-data-platform/connect-redshift-postgresql-alloydb.md Co-authored-by: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> --- .../connect-redshift-postgresql-alloydb.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/cloud/connect-data-platform/connect-redshift-postgresql-alloydb.md b/website/docs/docs/cloud/connect-data-platform/connect-redshift-postgresql-alloydb.md index d811457cad4..ddfd7c448cb 100644 --- a/website/docs/docs/cloud/connect-data-platform/connect-redshift-postgresql-alloydb.md +++ b/website/docs/docs/cloud/connect-data-platform/connect-redshift-postgresql-alloydb.md @@ -91,7 +91,7 @@ Bastion hosts might have additional SSH settings to disconnect inactive clients To prevent premature disconnections, you can adjust the settings on the bastion host: -- `ClientAliveCountMax 10` — Allows more checks before deciding the client is inactive. +- `ClientAliveCountMax ` — Configures the number of checks before deciding the client is inactive. For example, `ClientAliveCountMax 10` checks 10 times. - `ClientAliveInterval 30` — Checks for client activity every 30 seconds. These adjustments ensure that inactive SSH clients are disconnected after about 300 seconds, reducing the chance of tunnel failures. From 5f3829e6edb8a21511c8fed7b4bbd09958d9118c Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Fri, 1 Mar 2024 12:01:44 +0000 Subject: [PATCH 8/9] Update website/docs/docs/cloud/connect-data-platform/connect-redshift-postgresql-alloydb.md Co-authored-by: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> --- .../connect-redshift-postgresql-alloydb.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/cloud/connect-data-platform/connect-redshift-postgresql-alloydb.md b/website/docs/docs/cloud/connect-data-platform/connect-redshift-postgresql-alloydb.md index ddfd7c448cb..bea114e359b 100644 --- a/website/docs/docs/cloud/connect-data-platform/connect-redshift-postgresql-alloydb.md +++ b/website/docs/docs/cloud/connect-data-platform/connect-redshift-postgresql-alloydb.md @@ -92,7 +92,7 @@ Bastion hosts might have additional SSH settings to disconnect inactive clients To prevent premature disconnections, you can adjust the settings on the bastion host: - `ClientAliveCountMax ` — Configures the number of checks before deciding the client is inactive. For example, `ClientAliveCountMax 10` checks 10 times. -- `ClientAliveInterval 30` — Checks for client activity every 30 seconds. +- `ClientAliveInterval` — Configures when to check for client activity. For example, `ClientAliveInterval 30` checks every 30 seconds. These adjustments ensure that inactive SSH clients are disconnected after about 300 seconds, reducing the chance of tunnel failures. From 13d68a23bab0fbfb82cf41507d768afae9c5270e Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Fri, 1 Mar 2024 12:01:50 +0000 Subject: [PATCH 9/9] Update website/docs/docs/cloud/connect-data-platform/connect-redshift-postgresql-alloydb.md Co-authored-by: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> --- .../connect-redshift-postgresql-alloydb.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/cloud/connect-data-platform/connect-redshift-postgresql-alloydb.md b/website/docs/docs/cloud/connect-data-platform/connect-redshift-postgresql-alloydb.md index bea114e359b..ecf73ae81ec 100644 --- a/website/docs/docs/cloud/connect-data-platform/connect-redshift-postgresql-alloydb.md +++ b/website/docs/docs/cloud/connect-data-platform/connect-redshift-postgresql-alloydb.md @@ -93,6 +93,6 @@ To prevent premature disconnections, you can adjust the settings on the bastion - `ClientAliveCountMax ` — Configures the number of checks before deciding the client is inactive. For example, `ClientAliveCountMax 10` checks 10 times. - `ClientAliveInterval` — Configures when to check for client activity. For example, `ClientAliveInterval 30` checks every 30 seconds. -These adjustments ensure that inactive SSH clients are disconnected after about 300 seconds, reducing the chance of tunnel failures. +The example adjustments ensure that inactive SSH clients are disconnected after about 300 seconds, reducing the chance of tunnel failures.