Skip to content

Commit

Permalink
Merge branch 'current' into mirnawong1-patch-22
Browse files Browse the repository at this point in the history
  • Loading branch information
mirnawong1 authored Dec 5, 2023
2 parents 00dac80 + b5975a8 commit 65d2728
Show file tree
Hide file tree
Showing 25 changed files with 275 additions and 108 deletions.
18 changes: 3 additions & 15 deletions website/docs/docs/build/sl-getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,21 +74,9 @@ import SlSetUp from '/snippets/_new-sl-setup.md';

If you're encountering some issues when defining your metrics or setting up the dbt Semantic Layer, check out a list of answers to some of the questions or problems you may be experiencing.

<details>
<summary>How do I migrate from the legacy Semantic Layer to the new one?</summary>
<div>
<div>If you're using the legacy Semantic Layer, we highly recommend you <a href="https://docs.getdbt.com/docs/dbt-versions/upgrade-core-in-cloud">upgrade your dbt version </a> to dbt v1.6 or higher to use the new dbt Semantic Layer. Refer to the dedicated <a href="https://docs.getdbt.com/guides/sl-migration"> migration guide</a> for more info.</div>
</div>
</details>
<details>
<summary>How are you storing my data?</summary>
User data passes through the Semantic Layer on its way back from the warehouse. dbt Labs ensures security by authenticating through the customer's data warehouse. Currently, we don't cache data for the long term, but it might temporarily stay in the system for up to 10 minutes, usually less. In the future, we'll introduce a caching feature that allows us to cache data on our infrastructure for up to 24 hours.
</details>

<details>
<summary>Is the dbt Semantic Layer open source?</summary>
The dbt Semantic Layer is proprietary; however, some components of the dbt Semantic Layer are open source, such as dbt-core and MetricFlow. <br /><br />dbt Cloud Developer or dbt Core users can define metrics in their project, including a local dbt Core project, using the dbt Cloud IDE, dbt Cloud CLI, or dbt Core CLI. However, to experience the universal dbt Semantic Layer and access those metrics using the API or downstream tools, users must be on a dbt Cloud <a href="https://www.getdbt.com/pricing/">Team or Enterprise</a> plan. <br /><br />Refer to <a href="https://docs.getdbt.com/docs/cloud/billing">Billing</a> for more information.
</details>
import SlFaqs from '/snippets/_sl-faqs.md';

<SlFaqs/>


## Next steps
Expand Down
4 changes: 2 additions & 2 deletions website/docs/docs/cloud/about-cloud/regions-ip-addresses.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,8 @@ dbt Cloud is [hosted](/docs/cloud/about-cloud/architecture) in multiple regions

| Region | Location | Access URL | IP addresses | Developer plan | Team plan | Enterprise plan |
|--------|----------|------------|--------------|----------------|-----------|-----------------|
| North America multi-tenant [^1] | AWS us-east-1 (N. Virginia) | cloud.getdbt.com | 52.45.144.63 <br /> 54.81.134.249 <br />52.22.161.231 ||||
| North America Cell 1 [^1] | AWS us-east-1 (N.Virginia) | {account prefix}.us1.dbt.com | [Located in Account Settings](#locating-your-dbt-cloud-ip-addresses) ||||
| North America multi-tenant [^1] | AWS us-east-1 (N. Virginia) | cloud.getdbt.com | 52.45.144.63 <br /> 54.81.134.249 <br />52.22.161.231 <br />52.3.77.232 <br />3.214.191.130 <br />34.233.79.135 ||||
| North America Cell 1 [^1] | AWS us-east-1 (N. Virginia) | {account prefix}.us1.dbt.com | 52.45.144.63 <br /> 54.81.134.249 <br />52.22.161.231 <br />52.3.77.232 <br />3.214.191.130 <br />34.233.79.135 ||||
| EMEA [^1] | AWS eu-central-1 (Frankfurt) | emea.dbt.com | 3.123.45.39 <br /> 3.126.140.248 <br /> 3.72.153.148 ||||
| APAC [^1] | AWS ap-southeast-2 (Sydney)| au.dbt.com | 52.65.89.235 <br /> 3.106.40.33 <br /> 13.239.155.206 <br />||||
| Virtual Private dbt or Single tenant | Customized | Customized | Ask [Support](/community/resources/getting-help#dbt-cloud-support) for your IPs ||||
Expand Down
2 changes: 1 addition & 1 deletion website/docs/docs/collaborate/explore-projects.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: "Explore your dbt projects"
sidebar_label: "Explore dbt projects"
description: "Learn about dbt Explorer and how to interact with it to understand, improve, and leverage your data pipelines."
pagination_next: "docs/collaborate/explore-multiple-projects"
pagination_next: "docs/collaborate/model-performance"
pagination_prev: null
---

Expand Down
41 changes: 41 additions & 0 deletions website/docs/docs/collaborate/model-performance.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
---
title: "Model performance"
sidebar_label: "Model performance"
description: "Learn about ."
---

dbt Explorer provides metadata on dbt Cloud runs for in-depth model performance and quality analysis. This feature assists in reducing infrastructure costs and saving time for data teams by highlighting where to fine-tune projects and deployments &mdash; such as model refactoring or job configuration adjustments.

<LoomVideo id='98f33b3b7a374df0b7c04747eae6ef44' />

:::tip Public preview

The model performance feature is now available in dbt Explorer! Check it out!
:::

## The Performance overview page

You can pinpoint areas for performance enhancement by using the Performance overview page. This page presents a comprehensive analysis across all project models and displays the longest-running models, those most frequently executed, and the ones with the highest failure rates during runs/tests. Data can be segmented by environment and job type which can offer insights into:

- Most executed models (total count).
- Models with the longest execution time (average duration).
- Models with the most failures, detailing run failures (percentage and count) and test failures (percentage and count).

Each data point links to individual models in Explorer.

<Lightbox src="/img/docs/collaborate/dbt-explorer/example-performance-overview-page.png" width="80%" title="Example of Performance overview page"/>

You can view historical metadata for up to the past three months. Select the time horizon using the filter, which defaults to a two-week lookback.

<Lightbox src="/img/docs/collaborate/dbt-explorer/ex-2-week-default.png" title="Example of dropdown"/>

## The Model performance tab

You can view trends in execution times, counts, and failures by using the Model performance tab for historical performance analysis. Daily execution data includes:

- Average model execution time.
- Model execution counts, including failures/errors (total sum).

Clicking on a data point reveals a table listing all job runs for that day, with each row providing a direct link to the details of a specific run.

<Lightbox src="/img/docs/collaborate/dbt-explorer/example-model-performance-tab.png" title="Example of the Model performance tab"/>
50 changes: 50 additions & 0 deletions website/docs/docs/collaborate/project-recommendations.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
---
title: "Project recommendations"
sidebar_label: "Project recommendations"
description: "dbt Explorer provides recommendations that you can take to improve the quality of your dbt project."
---

:::tip Public preview

The project recommendations feature is now available in dbt Explorer! Check it out!

:::

dbt Explorer provides recommendations about your project from the `dbt_project_evaluator` [package](https://hub.getdbt.com/dbt-labs/dbt_project_evaluator/latest/) using metadata from the Discovery API.

Explorer also offers a global view, showing all the recommendations across the project for easy sorting and summarizing.

These recommendations provide insight into how you can build a more well documented, well tested, and well built project, leading to less confusion and more trust.

The Recommendations overview page includes two top-level metrics measuring the test and documentation coverage of the models in your project.

- **Model test coverage** &mdash; The percent of models in your project (models not from a package or imported via dbt Mesh) with at least one dbt test configured on them.
- **Model documentation coverage** &mdash; The percent of models in your project (models not from a package or imported via dbt Mesh) with a description.

<Lightbox src="/img/docs/collaborate/dbt-explorer/example-recommendations-overview.png" width="80%" title="Example of the Recommendations overview page with project metrics and the recommendations for all resources in the project"/>

## List of rules

| Category | Name | Description | Package Docs Link |
| --- | --- | --- | --- |
| Modeling | Direct Join to Source | Model that joins both a model and source, indicating a missing staging model | [GitHub](https://dbt-labs.github.io/dbt-project-evaluator/0.8/rules/modeling/#direct-join-to-source) |
| Modeling | Duplicate Sources | More than one source node corresponds to the same data warehouse relation | [GitHub](https://dbt-labs.github.io/dbt-project-evaluator/0.8/rules/modeling/#duplicate-sources) |
| Modeling | Multiple Sources Joined | Models with more than one source parent, indicating lack of staging models | [GitHub](https://dbt-labs.github.io/dbt-project-evaluator/0.8/rules/modeling/#multiple-sources-joined) |
| Modeling | Root Model | Models with no parents, indicating potential hardcoded references and need for sources | [GitHub](https://dbt-labs.github.io/dbt-project-evaluator/0.8/rules/modeling/#root-models) |
| Modeling | Source Fanout | Sources with more than one model child, indicating a need for staging models | [GitHub](https://dbt-labs.github.io/dbt-project-evaluator/0.8/rules/modeling/#source-fanout) |
| Modeling | Unused Source | Sources that are not referenced by any resource | [GitHub](https://dbt-labs.github.io/dbt-project-evaluator/0.8/rules/modeling/#unused-sources) |
| Performance | Exposure Dependent on View | Exposures with at least one model parent materialized as a view, indicating potential query performance issues | [GitHub](https://dbt-labs.github.io/dbt-project-evaluator/0.8/rules/performance/#exposure-parents-materializations) |
| Testing | Missing Primary Key Test | Models with insufficient testing on the grain of the model. | [GitHub](https://dbt-labs.github.io/dbt-project-evaluator/0.8/rules/testing/#missing-primary-key-tests) |
| Documentation | Undocumented Models | Models without a model-level description | [GitHub](https://dbt-labs.github.io/dbt-project-evaluator/0.8/rules/documentation/#undocumented-models) |
| Documentation | Undocumented Source | Sources (collections of source tables) without descriptions | [GitHub](https://dbt-labs.github.io/dbt-project-evaluator/0.8/rules/documentation/#undocumented-sources) |
| Documentation | Undocumented Source Tables | Source tables without descriptions | [GitHub](https://dbt-labs.github.io/dbt-project-evaluator/0.8/rules/documentation/#undocumented-source-tables) |
| Governance | Public Model Missing Contract | Models with public access that do not have a model contract to ensure the data types | [GitHub](https://dbt-labs.github.io/dbt-project-evaluator/0.8/rules/governance/#public-models-without-contracts) |


## The Recommendations tab

Models, sources and exposures each also have a Recommendations tab on their resource details page, with the specific recommendations that correspond to that resource:

<Lightbox src="/img/docs/collaborate/dbt-explorer/example-recommendations-tab.png" width="80%" title="Example of the Recommendations tab "/>


Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ dbt Core v1.6 has three significant areas of focus:
## Resources

- [Changelog](https://github.com/dbt-labs/dbt-core/blob/1.6.latest/CHANGELOG.md)
- [CLI Installation guide](/docs/core/installation-overview
- [dbt Core installation guide](/docs/core/installation-overview)
- [Cloud upgrade guide](/docs/dbt-versions/upgrade-core-in-cloud)
- [Release schedule](https://github.com/dbt-labs/dbt-core/issues/7481)

Expand Down
2 changes: 2 additions & 0 deletions website/docs/docs/use-dbt-semantic-layer/gsheets.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@ The dbt Semantic Layer offers a seamless integration with Google Sheets through
- You have a Google account with access to Google Sheets.
- You can install Google add-ons.
- You have a dbt Cloud Environment ID and a [service token](/docs/dbt-cloud-apis/service-tokens) to authenticate with from a dbt Cloud account.
- You must have a dbt Cloud Team or Enterprise [account](https://www.getdbt.com/pricing). Suitable for both Multi-tenant and Single-tenant deployment.
- Single-tenant accounts should contact their account representative for necessary setup and enablement.

## Installing the add-on

Expand Down
19 changes: 4 additions & 15 deletions website/docs/docs/use-dbt-semantic-layer/quickstart-sl.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ MetricFlow, a powerful component of the dbt Semantic Layer, simplifies the creat
Use this guide to fully experience the power of the universal dbt Semantic Layer. Here are the following steps you'll take:

- [Create a semantic model](#create-a-semantic-model) in dbt Cloud using MetricFlow
- [Define metrics](#define-metrics) in dbt Cloud using MetricFlow
- [Define metrics](#define-metrics) in dbt using MetricFlow
- [Test and query metrics](#test-and-query-metrics) with MetricFlow
- [Run a production job](#run-a-production-job) in dbt Cloud
- [Set up dbt Semantic Layer](#setup) in dbt Cloud
Expand Down Expand Up @@ -88,20 +88,9 @@ import SlSetUp from '/snippets/_new-sl-setup.md';

If you're encountering some issues when defining your metrics or setting up the dbt Semantic Layer, check out a list of answers to some of the questions or problems you may be experiencing.

<details>
<summary>How do I migrate from the legacy Semantic Layer to the new one?</summary>
<div>
<div>If you're using the legacy Semantic Layer, we highly recommend you <a href="https://docs.getdbt.com/docs/dbt-versions/upgrade-core-in-cloud">upgrade your dbt version </a> to dbt v1.6 or higher to use the new dbt Semantic Layer. Refer to the dedicated <a href="https://docs.getdbt.com/guides/sl-migration"> migration guide</a> for more info.</div>
</div>
</details>
<details>
<summary>How are you storing my data?</summary>
User data passes through the Semantic Layer on its way back from the warehouse. dbt Labs ensures security by authenticating through the customer's data warehouse. Currently, we don't cache data for the long term, but it might temporarily stay in the system for up to 10 minutes, usually less. In the future, we'll introduce a caching feature that allows us to cache data on our infrastructure for up to 24 hours.
</details>
<details>
<summary>Is the dbt Semantic Layer open source?</summary>
The dbt Semantic Layer is proprietary; however, some components of the dbt Semantic Layer are open source, such as dbt-core and MetricFlow. <br /><br />dbt Cloud Developer or dbt Core users can define metrics in their project, including a local dbt Core project, using the dbt Cloud IDE, dbt Cloud CLI, or dbt Core CLI. However, to experience the universal dbt Semantic Layer and access those metrics using the API or downstream tools, users must be on a dbt Cloud <a href="https://www.getdbt.com/pricing/">Team or Enterprise</a> plan. <br /><br />Refer to <a href="https://docs.getdbt.com/docs/cloud/billing">Billing</a> for more information.
</details>
import SlFaqs from '/snippets/_sl-faqs.md';

<SlFaqs/>


## Next steps
Expand Down
53 changes: 23 additions & 30 deletions website/docs/docs/use-dbt-semantic-layer/sl-architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,45 +14,38 @@ The dbt Semantic Layer allows you to define metrics and use various interfaces t

<Lightbox src="/img/docs/dbt-cloud/semantic-layer/sl-architecture.jpg" width="85%" title="The diagram displays how your data flows using the dbt Semantic Layer and the variety of integration tools it supports."/>



## dbt Semantic Layer components
## Components

The dbt Semantic Layer includes the following components:


| Components | Information | dbt Core users | Developer plans | Team plans | Enterprise plans | License |
| --- | --- | :---: | :---: | :---: | --- |
| --- | --- | :---: | :---: | :---: | :---: |
| **[MetricFlow](/docs/build/about-metricflow)** | MetricFlow in dbt allows users to centrally define their semantic models and metrics with YAML specifications. ||||| BSL package (code is source available) |
| **MetricFlow Server**| A proprietary server that takes metric requests and generates optimized SQL for the specific data platform. ||||| Proprietary, Cloud (Team & Enterprise)|
| **Semantic Layer Gateway** | A service that passes queries to the MetricFlow server and executes the SQL generated by MetricFlow against the data platform| <br></br>|||| Proprietary, Cloud (Team & Enterprise) |
| **Semantic Layer APIs** | The interfaces allow users to submit metric queries using GraphQL and JDBC APIs. They also serve as the foundation for building first-class integrations with various tools. ||||| Proprietary, Cloud (Team & Enterprise)|
| **dbt Semantic interfaces**| A configuration spec for defining metrics, dimensions, how they link to each other, and how to query them. The [dbt-semantic-interfaces](https://github.com/dbt-labs/dbt-semantic-interfaces) is available under Apache 2.0. ||||| Proprietary, Cloud (Team & Enterprise)|
| **Service layer** | Coordinates query requests and dispatching the relevant metric query to the target query engine. This is provided through dbt Cloud and is available to all users on dbt version 1.6 or later. The service layer includes a Gateway service for executing SQL against the data platform. | || || Proprietary, Cloud (Team & Enterprise) |
| **[Semantic Layer APIs](/docs/dbt-cloud-apis/sl-api-overview)** | The interfaces allow users to submit metric queries using GraphQL and JDBC APIs. They also serve as the foundation for building first-class integrations with various tools. ||||| Proprietary, Cloud (Team & Enterprise)|


## Related questions
## Feature comparison

<details>
<summary>How do I migrate from the legacy Semantic Layer to the new one?</summary>
<div>
<div>If you're using the legacy Semantic Layer, we highly recommend you <a href="https://docs.getdbt.com/docs/dbt-versions/upgrade-core-in-cloud">upgrade your dbt version </a> to dbt v1.6 or higher to use the new dbt Semantic Layer. Refer to the dedicated <a href="https://docs.getdbt.com/guides/sl-migration"> migration guide</a> for more info.</div>
</div>
</details>

<details>
<summary>How are you storing my data?</summary>
User data passes through the Semantic Layer on its way back from the warehouse. dbt Labs ensures security by authenticating through the customer's data warehouse. Currently, we don't cache data for the long term, but it might temporarily stay in the system for up to 10 minutes, usually less. In the future, we'll introduce a caching feature that allows us to cache data on our infrastructure for up to 24 hours.
</details>
<details>
<summary>Is the dbt Semantic Layer open source?</summary>
The dbt Semantic Layer is proprietary; however, some components of the dbt Semantic Layer are open source, such as dbt-core and MetricFlow. <br /><br />dbt Cloud Developer or dbt Core users can define metrics in their project, including a local dbt Core project, using the dbt Cloud IDE, dbt Cloud CLI, or dbt Core CLI. However, to experience the universal dbt Semantic Layer and access those metrics using the API or downstream tools, users must be on a dbt Cloud <a href="https://www.getdbt.com/pricing/">Team or Enterprise</a> plan. <br /><br />Refer to <a href="https://docs.getdbt.com/docs/cloud/billing">Billing</a> for more information.
</details>
<details>
<summary>Is there a dbt Semantic Layer discussion hub?</summary>
<div>
<div>Yes absolutely! Join the <a href="https://getdbt.slack.com">dbt Slack community</a> and <a href="https://getdbt.slack.com/archives/C046L0VTVR6">#dbt-cloud-semantic-layer slack channel</a> for all things related to the dbt Semantic Layer.
</div>
</div>
</details>
The following table compares the features available in dbt Cloud and source available in dbt Core:

| Feature | MetricFlow Source available | dbt Semantic Layer with dbt Cloud |
| ----- | :------: | :------: |
| Define metrics and semantic models in dbt using the MetricFlow spec |||
| Generate SQL from a set of config files |||
| Query metrics and dimensions through the command line interface (CLI) |||
| Query dimension, entity, and metric metadata through the CLI |||
| Query metrics and dimensions through semantic APIs (ADBC, GQL) |||
| Connect to downstream integrations (Tableau, Hex, Mode, Google Sheets, and so on.) |||
| Create and run Exports to save metrics queries as tables in your data platform. || Coming soon |

## FAQs

import SlFaqs from '/snippets/_sl-faqs.md';

<SlFaqs/>

</VersionBlock>

Expand Down
Loading

0 comments on commit 65d2728

Please sign in to comment.