Skip to content

Commit

Permalink
Merge branch 'current' into BrJan-patch-4
Browse files Browse the repository at this point in the history
  • Loading branch information
mirnawong1 authored Jan 25, 2024
2 parents 3d85728 + b05c4f4 commit 475a73e
Show file tree
Hide file tree
Showing 16 changed files with 100 additions and 69 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -21,13 +21,11 @@ We're not limited to just passing measures through to our metrics, we can also _

```YAML
- name: food_revenue
description: The revenue from food in each order.
label: Food Revenue
type: simple
type_params:
measure: revenue
filter: |
{{ Dimension('order__is_food_order') }} = true
description: The revenue from food in each order.
label: Food Revenue
type: simple
type_params:
measure: food_revenue
```
- 📝 Now we can set up our ratio metric.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,10 @@ id: 1-how-we-style-our-dbt-models
- 👥 Models should be pluralized, for example, `customers`, `orders`, `products`.
- 🔑 Each model should have a primary key.
- 🔑 The primary key of a model should be named `<object>_id`, for example, `account_id`. This makes it easier to know what `id` is being referenced in downstream joined models.
- Use underscores for naming dbt models; avoid dots.
-`models_without_dots`
-`models.with.dots`
- Most data platforms use dots to separate `database.schema.object`, so using underscores instead of dots reduces your need for [quoting](/reference/resource-properties/quoting) as well as the risk of issues in certain parts of dbt Cloud. For more background, refer to [this GitHub issue](https://github.com/dbt-labs/dbt-core/issues/3246).
- 🔑 Keys should be string data types.
- 🔑 Consistency is key! Use the same field names across models where possible. For example, a key to the `customers` table should be named `customer_id` rather than `user_id` or 'id'.
- ❌ Do not use abbreviations or aliases. Emphasize readability over brevity. For example, do not use `cust` for `customer` or `o` for `orders`.
Expand Down
2 changes: 0 additions & 2 deletions website/docs/docs/build/models.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,6 @@ pagination_next: "docs/build/sql-models"
pagination_prev: null
---

## Overview

dbt Core and Cloud are composed of different moving parts working harmoniously. All of them are important to what dbt does — transforming data—the 'T' in ELT. When you execute `dbt run`, you are running a model that will transform your data without that data ever leaving your warehouse.

Models are where your developers spend most of their time within a dbt environment. Models are primarily written as a `select` statement and saved as a `.sql` file. While the definition is straightforward, the complexity of the execution will vary from environment to environment. Models will be written and rewritten as needs evolve and your organization finds new ways to maximize efficiency.
Expand Down
5 changes: 4 additions & 1 deletion website/docs/docs/build/sql-models.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,10 @@ dbt's Python capabilities are an extension of its capabilities with SQL models.
A SQL model is a `select` statement. Models are defined in `.sql` files (typically in your `models` directory):
- Each `.sql` file contains one model / `select` statement
- The model name is inherited from the filename.
- Models can be nested in subdirectories within the `models` directory
- We strongly recommend using underscores for model names, not dots. For example, use `models/my_model.sql` instead of `models/my.model.sql`.
- Models can be nested in subdirectories within the `models` directory.

Refer to [How we style our dbt models](/best-practices/how-we-style/1-how-we-style-our-dbt-models) for details on how we recommend you name your models.

When you execute the [`dbt run` command](/reference/commands/run), dbt will build this model <Term id="data-warehouse" /> by wrapping it in a `create view as` or `create table as` statement.

Expand Down
54 changes: 24 additions & 30 deletions website/docs/docs/cloud/dbt-cloud-ide/develop-in-the-cloud.md
Original file line number Diff line number Diff line change
Expand Up @@ -133,38 +133,32 @@ The dbt Cloud IDE makes it possible to [build and view](/docs/collaborate/build-

## Related docs

- [dbt Cloud features](/docs/cloud/about-cloud/dbt-cloud-features)
- [IDE user interface](/docs/cloud/dbt-cloud-ide/ide-user-interface)
- [How we style our dbt projects](/best-practices/how-we-style/0-how-we-style-our-dbt-projects)
- [User interface](/docs/cloud/dbt-cloud-ide/ide-user-interface)
- [Version control basics](/docs/collaborate/git/version-control-basics)
- [dbt Commands](/reference/dbt-commands)
- [dbt Cloud IDE release notes](/tags/ide)

## Related questions

<FAQ path="Git/gitignore"/> <br />

<details>
<summary>Is there a cost to using the Cloud IDE?</summary>
<div>
<div>Not at all! You can use dbt Cloud when you sign up for the <a href="https://www.getdbt.com/pricing/">Free Developer plan</a>, which comes with one developer seat. If you’d like to access more features or have more developer seats, you can upgrade your account to the Team or Enterprise plan. See <a href="https://www.getdbt.com/pricing/">dbt pricing plans</a> for more details. </div>
</div>
</details>
<details>
<summary>Can I be a contributor to dbt Cloud?</summary>
<div>
<div>Anyone can contribute to the dbt project. And whether it's a dbt package, a plugin, dbt-core, or this documentation site, contributing to the open-source code that supports the dbt ecosystem is a great way to level yourself up as a developer, and give back to the community. See <a href="https://docs.getdbt.com/community/resources/oss-expectations">Contributing</a> for details on what to expect when contributing to the dbt open source software (OSS). </div>
</div>
</details>
<details>
<summary>What is the difference between developing on the dbt Cloud IDE, the dbt Cloud CLI, and dbt Core?</summary>
<div>
<div>You can develop dbt using the web-based IDE in dbt Cloud or on the command line interface using the dbt Cloud CLI or open-source dbt Core, all of which enable you to execute dbt commands. The key distinction between the dbt Cloud CLI and dbt Core is the dbt Cloud CLI is tailored for dbt Cloud's infrastructure and integrates with all its features.<br /><br />
<span>&mdash; </span>
<b>dbt Cloud IDE:</b> <a href="https://docs.getdbt.com/docs/cloud/about-cloud/dbt-cloud-features">dbt Cloud</a> is a web-based application that allows you to develop dbt projects with the IDE, includes a purpose-built scheduler, and provides an easier way to share your dbt documentation with your team. The IDE is a faster and more reliable way to deploy your dbt models and provides a real-time editing and execution environment for your dbt project. <br /><br />
<span>&mdash; </span>
<b>dbt Cloud CLI: </b> <a href="https://docs.getdbt.com/docs/cloud/cloud-cli-installation">The dbt Cloud CLI</a> allows you to run dbt commands against your dbt Cloud development environment from your local command line or code editor. It supports cross-project ref, speedier, lower-cost builds, automatic deferral of build artifacts, and more. <br /><br />
<span>&mdash; </span>
<b>dbt Core:</b> dbt Core is an <a href="https://github.com/dbt-labs/dbt">open-sourced</a> software that’s freely available. You can build your dbt project in a code editor, and run dbt commands from the command line.
</div>
</div>
</details>
<FAQ path="Git/gitignore"/>

<detailsToggle alt_header="Is there a cost to using the Cloud IDE?">
Not at all! You can use dbt Cloud when you sign up for the <a href="https://www.getdbt.com/pricing/">Free Developer plan</a>, which comes with one developer seat. If you’d like to access more features or have more developer seats, you can upgrade your account to the Team or Enterprise plan.<br />

Refer to <a href="https://www.getdbt.com/pricing/">dbt pricing plans</a> for more details.
</detailsToggle>

<detailsToggle alt_header="Can I be a contributor to dbt Cloud?">
As a proprietary product, dbt Cloud's source code isn't available for community contributions. If you want to build something in the dbt ecosystem, we encourage you to review [this article](/community/contributing/contributing-coding) about contributing to a dbt package, a plugin, dbt-core, or this documentation site. Participation in open-source is a great way to level yourself up as a developer, and give back to the community.
</detailsToggle>

<detailsToggle alt_header="What is the difference between developing on the dbt Cloud IDE, the dbt Cloud CLI, and dbt Core?">
You can develop dbt using the web-based IDE in dbt Cloud or on the command line interface using the dbt Cloud CLI or open-source dbt Core, all of which enable you to execute dbt commands. The key distinction between the dbt Cloud CLI and dbt Core is the dbt Cloud CLI is tailored for dbt Cloud's infrastructure and integrates with all its features:

- dbt Cloud IDE: <a href="https://docs.getdbt.com/docs/cloud/about-cloud/dbt-cloud-features">dbt Cloud</a> is a web-based application that allows you to develop dbt projects with the IDE, includes a purpose-built scheduler, and provides an easier way to share your dbt documentation with your team. The IDE is a faster and more reliable way to deploy your dbt models and provides a real-time editing and execution environment for your dbt project.

- dbt Cloud CLI: <a href="https://docs.getdbt.com/docs/cloud/cloud-cli-installation">The dbt Cloud CLI</a> allows you to run dbt commands against your dbt Cloud development environment from your local command line or code editor. It supports cross-project ref, speedier, lower-cost builds, automatic deferral of build artifacts, and more.

- dbt Core: dbt Core is an <a href="https://github.com/dbt-labs/dbt">open-sourced</a> software that’s freely available. You can build your dbt project in a code editor, and run dbt commands from the command line

</detailsToggle>
10 changes: 8 additions & 2 deletions website/docs/docs/cloud/secure/postgres-privatelink.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,13 +49,13 @@ On the provisioned VPC endpoint service, click the **Allow principals** tab. Cli

- Principal: `arn:aws:iam::346425330055:role/MTPL_Admin`

<Lightbox src="/img/docs/dbt-cloud/redshiftprivatelink5.png" width="70%" title="Enter ARN"/>
<Lightbox src="/img/docs/dbt-cloud/privatelink-allow-principals.png" width="70%" title="Enter ARN"/>

### 3. Obtain VPC Endpoint Service Name

Once the VPC Endpoint Service is provisioned, you can find the service name in the AWS console by navigating to **VPC****Endpoint Services** and selecting the appropriate endpoint service. You can copy the service name field value and include it in your communication to dbt Cloud support.

<Lightbox src="/img/docs/dbt-cloud/redshiftprivatelink6.png" width="70%" title="Get service name field value"/>
<Lightbox src="/img/docs/dbt-cloud/privatelink-endpoint-service-name.png" width="70%" title="Get service name field value"/>

### 4. Add the required information to the template below, and submit your request to [dbt Support](https://docs.getdbt.com/community/resources/getting-help#dbt-cloud-support):
```
Expand All @@ -68,6 +68,12 @@ Subject: New Multi-Tenant PrivateLink Request

dbt Labs will work on your behalf to complete the PrivateLink setup. Please allow 1-2 business days for this process to complete. Support will contact you when the endpoint is available.

### 5. Accepting the connection request

When you have been notified that the resources are provisioned within the dbt Cloud environment, you must accept the endpoint connection (unless the VPC Endpoint Service is set to auto-accept connection requests). Requests can be accepted through the AWS console, as seen below, or through the AWS CLI.

<Lightbox src="/img/docs/dbt-cloud/cloud-configuring-dbt-cloud/accept-request.png" width="80%" title="Accept the connection request" />

## Create Connection in dbt Cloud

Once dbt Cloud support completes the configuration, you can start creating new connections using PrivateLink.
Expand Down
4 changes: 2 additions & 2 deletions website/docs/docs/cloud/secure/redshift-privatelink.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,13 +85,13 @@ On the provisioned VPC endpoint service, click the **Allow principals** tab. Cli

- Principal: `arn:aws:iam::346425330055:role/MTPL_Admin`

<Lightbox src="/img/docs/dbt-cloud/redshiftprivatelink5.png" title="Enter ARN"/>
<Lightbox src="/img/docs/dbt-cloud/privatelink-allow-principals.png" title="Enter ARN"/>

### 3. Obtain VPC Endpoint Service Name

Once the VPC Endpoint Service is provisioned, you can find the service name in the AWS console by navigating to **VPC****Endpoint Services** and selecting the appropriate endpoint service. You can copy the service name field value and include it in your communication to dbt Cloud support.

<Lightbox src="/img/docs/dbt-cloud/redshiftprivatelink6.png" title="Get service name field value"/>
<Lightbox src="/img/docs/dbt-cloud/privatelink-endpoint-service-name.png" title="Get service name field value"/>

### 4. Add the required information to the template below, and submit your request to [dbt Support](https://docs.getdbt.com/community/resources/getting-help#dbt-cloud-support):
```
Expand Down
50 changes: 37 additions & 13 deletions website/docs/docs/cloud/secure/vcs-privatelink.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,8 +23,11 @@ Creating an Interface VPC PrivateLink connection requires creating multiple AWS

This approach is distinct from and does not require you to implement VPC peering between your AWS account(s) and dbt Cloud.

You need these resource to create a PrivateLink connection, which allows the dbt Cloud application to connect to your self-hosted cloud VCS. These resources can be created via the AWS Console, AWS CLI, or Infrastructure-as-Code such as [Terraform](https://registry.terraform.io/providers/hashicorp/aws/latest/docs) or [AWS CloudFormation](https://aws.amazon.com/cloudformation/).
### 1. Provision AWS resources

Creating an Interface VPC PrivateLink connection requires creating multiple AWS resources in the account containing, or connected to, your self-hosted cloud VCS. These resources can be created via the AWS Console, AWS CLI, or Infrastructure-as-Code such as [Terraform](https://registry.terraform.io/providers/hashicorp/aws/latest/docs) or [AWS CloudFormation](https://aws.amazon.com/cloudformation/).

- **Security Group (AWS hosted only)** &mdash; If you are connecting to an existing VCS install, this likely already exists, however, you may need to add or modify Security Group rules to accept traffic from the Network Load Balancer (NLB) created for this Endpoint Service.
- **Target Group(s)** - A [Target Group](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-target-groups.html) is attached to a [Listener](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-listeners.html) on the NLB and is responsible for routing incoming requests to healthy targets in the group. If connecting to the VCS system over both SSH and HTTPS, two **Target Groups** will need to be created.
- **Target Type (choose most applicable):**
- **Instance/ASG:** Select existing EC2 instance(s) where the VCS system is running, or [an autoscaling group](https://docs.aws.amazon.com/autoscaling/ec2/userguide/attach-load-balancer-asg.html) (ASG) to automatically attach any instances launched from that ASG.
Expand All @@ -46,23 +49,43 @@ You need these resource to create a PrivateLink connection, which allows the dbt
- **Load balancer:** Attach the NLB created in the previous step.
- **Acceptance required (recommended)**: When enabled, requires a new connection request to the VPC Endpoint Service to be accepted by the customer before connectivity is allowed ([details](https://docs.aws.amazon.com/vpc/latest/privatelink/configure-endpoint-service.html#accept-reject-connection-requests)).

Once these resources have been provisioned, access needs to be granted for the dbt Labs AWS account to create a VPC Endpoint in our VPC. On the newly created VPC Endpoint Service, add a new [Allowed Principal](https://docs.aws.amazon.com/vpc/latest/privatelink/configure-endpoint-service.html#add-remove-permissions) for the appropriate dbt Labs principal:
### 2. Grant dbt AWS account access to the VPC Endpoint Service

- **AWS Account ID:** `arn:aws:iam::<account id>:root` (contact your dbt Labs account representative for appropriate account ID).
Once these resources have been provisioned, access needs to be granted for the dbt Labs AWS account to create a VPC Endpoint in our VPC. On the provisioned VPC endpoint service, click the **Allow principals** tab. Click **Allow principals** to grant access. Enter the ARN of the following IAM role in the appropriate production AWS account and save your changes ([details](https://docs.aws.amazon.com/vpc/latest/privatelink/configure-endpoint-service.html#add-remove-permissions)).

### Completing the connection
- Principal: `arn:aws:iam::346425330055:role/MTPL_Admin`

To complete the connection, dbt Labs must now provision a VPC Endpoint to connect to your VPC Endpoint Service. This requires you send the following information:

- VPC Endpoint Service name:
<Lightbox src="/img/docs/dbt-cloud/privatelink-allow-principals.png" width="70%" title="Enter ARN"/>

<Lightbox src="/img/docs/dbt-cloud/cloud-configuring-dbt-cloud/vpc-endpoint-service-name.png" width="80%" title="Location of the VPC Endpoint Service name in the AWS console" />
### 3. Obtain VPC Endpoint Service Name

- **DNS configuration:** If the connection to the VCS service requires a custom domain and/or URL for TLS, a private hosted zone can be configured by the dbt Labs Infrastructure team in the dbt Cloud private network. For example:
- **Private hosted zone:** `examplecorp.com`
- **DNS record:** `github.examplecorp.com`
Once the VPC Endpoint Service is provisioned and configured find the service name in the AWS console by navigating to **VPC****Endpoint Services** and selecting the appropriate endpoint service. Copy the service name field value and include it in your communication to dbt Cloud support.

### Accepting the connection request
<Lightbox src="/img/docs/dbt-cloud/privatelink-endpoint-service-name.png" width="70%" title="Get service name field value"/>

:::note Custom DNS configuration

If the connection to the VCS service requires a custom domain and/or URL for TLS, a private hosted zone can be configured by the dbt Labs Infrastructure team in the dbt Cloud private network. For example:
- Private hosted zone: examplecorp.com
- DNS record: github.examplecorp.com

:::

### 4. Add the required information to the template below, and submit your request to [dbt Support](https://docs.getdbt.com/community/resources/getting-help#dbt-cloud-support):
```
Subject: New Multi-Tenant PrivateLink Request
- Type: VCS Interface-type
- VPC Endpoint Service Name:
- Custom DNS (optional)
- Private hosted zone:
- DNS record:
- VCS install AWS Region (e.g., us-east-1, eu-west-2):
- dbt Cloud multi-tenant environment (US, EMEA, AU):
```

dbt Labs will work on your behalf to complete the PrivateLink setup. Please allow 1-2 business days for this process to complete. Support will contact you when the endpoint is available.

### 5. Accepting the connection request

When you have been notified that the resources are provisioned within the dbt Cloud environment, you must accept the endpoint connection (unless the VPC Endpoint Service is set to auto-accept connection requests). Requests can be accepted through the AWS console, as seen below, or through the AWS CLI.

Expand All @@ -73,10 +96,11 @@ Once you accept the endpoint connection request, you can use the PrivateLink end
## Configure in dbt Cloud

Once dbt confirms that the PrivateLink integration is complete, you can use it in a new or existing git configuration.

1. Select **PrivateLink Endpoint** as the connection type, and your configured integrations will appear in the dropdown menu.
2. Select the configured endpoint from the drop down list.
3. Click **Save**.

<Lightbox src="/img/docs/dbt-cloud/cloud-configuring-dbt-cloud/vcs-setup-new.png" width="80%" title="Configuring a new git integration with PrivateLink" />

<Lightbox src="/img/docs/dbt-cloud/cloud-configuring-dbt-cloud/vcs-setup-existing.png" width="80%" title="Editing an existing git integration with PrivateLink" />
<Lightbox src="/img/docs/dbt-cloud/cloud-configuring-dbt-cloud/vcs-setup-existing.png" width="80%" title="Editing an existing git integration with PrivateLink" />
Loading

0 comments on commit 475a73e

Please sign in to comment.