The DSE team uses Terraform to manage cloud infrastructure. +Our stack includes:
+flowchart TD
+ subgraph AWS
+ J[GitHub CD\nbot user]
+ G[Artifact in S3]
+ subgraph VPC
+ subgraph Managed Airflow
+ K1[Scheduler]
+ K2[Worker]
+ K3[Webserver]
+ end
+ F[AWS Batch Job\n on Fargate]
+ end
+ E[AWS ECR Docker\nRepository]
+ end
+ subgraph GitHub
+ A[Code Repository]
+ end
+ E --> F
+ A -- Code quality check\n GitHub action --> A
+ A -- Job submission\nvia GitHub Action --> F
+ A -- Docker build \nGitHub Action --> E
+ A --> H[CalData\nadministrative\nuser]
+ H -- Terraform -----> AWS
+ K2 -- Job submission\nvia Airflow --> F
+ K1 <--> K2
+ K3 <--> K1
+ K3 <--> K2
+ F --> G
+ J -- Bot Credentials --> AThis project requires Terraform to run. +You might use a different package manager to install it depending on your system.
+For Macs, you can use brew:
brew install terraform
+Anaconda users on any architecture should be able to use conda or mamba:
conda install -c conda-forge terraform
+We also use tflint for linting, and terraform-docs to help with documentation of resources.
+These can be installed in the same manner, e.g.:
conda install -c conda-forge tflint go-terraform-docs
+There are a number of pre-commit checks that run on committing as well as in CI. +To install the checks, run the following from the repository root:
+pre-commit install
+You can manually run the pre-commit checks using:
+pre-commit run --all-files
+When deploying a new version of your infrastrucutre, Terraform diffs the current state +against what you have specified in your infrastructure-as-code. +The current state is tracked in a JSON document, +which can be stored in any of a number of locations (including local files).
+This project stores remote state using the S3 backend.
+Different applications or environments can be isolated from each other by using
+different S3 buckets for holding their state.
+We reuse a terraform configuration (terraform/s3-remote-state) for setting up the S3 backend
Note
+The S3 remote state configuration is not a proper module because it contains
+a provider block. Different deployments of the configuration are controlled
+by giving it different tfvars files, and capturing the outputs for use in
+a tfbackend file.
Here is an example set of commands for bootstrapping a new S3 backend for a deployment. +Suppose the deployment is a QA environment of our Snowflake project:
+cd terraform/snowflake/environments/qa # Go to the new environment directory
+mkdir remote-state # Create a remote-state directory
+cd remote-state
+ln -s ../../../s3-remote-state/main.tf main.tf # symlink the s3 configuration
+terraform init # initialize the remote state backend
+terraform apply -var="owner=dse" -var="environment=qa" -var="project=snowflake" # Create the infrastructure
+terraform output > ../dse-snowflake-qa.tfbackend # Pipe the outputs to a .tfbackend
+
+cd ..
+terraform init -backend-config=./dse-snowflake-qa.tfbackend # Configure the deployment with the new backend.
+When you are ready to deploy a new version of the infrastructure, run
+terraform apply
+This will output the changes to the infrastructure that will be made, and prompt you for confirmation.
+Terraform deployments include a lockfile with hashes of installed packages. +Because we have mixed development environments (i.e., Macs locally, Linux in CI), +it is helpful to include both Mac and Linux builds of terraform packages in the lockfile. +This needs to be done every time package versions are updated:
+terraform init -upgrade # Upgrade versions
+terraform providers lock -platform=linux_amd64 -platform=darwin_amd64 # include Mac and Linux binaries
+| Name | +Version | +
|---|---|
| terraform | +>= 1.0 | +
| aws | +4.56.0 | +
| random | +3.4.3 | +
| Name | +Version | +
|---|---|
| aws | +4.56.0 | +
| random | +3.4.3 | +
No modules.
+| Name | +Description | +Type | +Default | +Required | +
|---|---|---|---|---|
| environment | +Deployment environment of the resource | +string |
+"dev" |
+no | +
| owner | +Owner of the resource | +string |
+"dse" |
+no | +
| project | +Name of the project the resource is serving | +string |
+"infra" |
+no | +
| region | +Region for AWS resources | +string |
+"us-west-2" |
+no | +
| snowflake_loader_secret | +ARN for SecretsManager login info to Snowflake with loader role | +string |
+null |
+no | +
| Name | +Description | +
|---|---|
| state | +Resources from terraform-state | +
This page documents the Data Services and Engineering Team's practices around GitHub-based development and code review.
+The process for GitHub-based development is:
+main:
+ git switch -c <branch-name>
+git push <remote-name> <branch-name>
+main.main now.We recommend reading "Have empathy on pull requests" +and "How about code reviews?" +as nice references for how to be empathetic as the opener of a pull request.
+In particular, it's important to remember that you are the subject matter expert for a PR. +The reviewer will likely not know anything about the path you took to a particular solution, +what approaches did not work, and what tradeoffs you encountered. +It's your job to communicate that context for reviewers to help them review your code. +This can include comments in the GitHub UI, comments in the code base, and even self-reviews.
+Making use of code linters and formatters helps to establish a consistent style for a project +and removes a whole whole class of common errors and disagreements. +Even if one can take issue with specific conventions or rules, +having them used consistently within a team pays big dividends over time.
+Many of our projects use pre-commit to enforce the linter and formatter conventions.
+To set up your pre-commit environment locally (requires a Python development environment, run
pre-commit install
+The next time you make a commit, the pre-commit hooks will run on the contents of your commit +(the first time may be a bit slow as there is some additional setup).
+More merge commits in a PR can make review more difficult, +as contents from unrelated work can appear in the code diff. +Sometimes they are necessary for particularly large or long-running branches, +but for most work you should try to avoid them. +The following guidelines can help:
+maingit rebase can help keep a feature branch easy to review and reason about.If your pull request adds any new features or changes any workflows for users +of the project, you should include documentation. +Otherwise, the hard work you did to implement a feature may go unnoticed/unused! +What this looks like will vary from project to project, but might include:
+New functionality ideally should have automated tests. +As with documentation, these tests will look different depending upon the project needs. +A good test will:
+One nice strategy for bugfixes is to write a failing test before making a fix, +then verifying that the fix makes the test pass. +It is surprisingly common for tests to accidentally not cover the behavior they are intended to.
+As above, reviewers should have empathy for the author of a pull request. +You as a reviewer are unaware of the constraints and tradeoffs that an author might have encountered.
+Some general tips for conducting productive reviews:
+Before merging a pull request, maintainers should make every effort to ensure that CI passes. +Often this will require looking into the logs of a failed run to see what went wrong, +and alerting the pull request author. +Ideally, no pull request should be merged if there are CI failures, +as broken CI in main can easily mask problems with other PRs, +and a consistently broken CI can be demoralizing for maintainers.
+However, in practice, there are occasionally flaky tests, +broken upstream dependencies, and failures that are otherwise obviously not related to the PR at hand. +If that is the case, a reviewer may merge a PR with failing tests, +but they should be prepared to follow up with any failures that result from such an unsafe operation.
+Note: these conventions and recommendations are partially drawn from maintainer guidelines for +JupyterHub and +Dask.
+ + + + + + +GitHub Codespaces allow you to spin up an ephemeral development environment in VS Code +which includes a git repository, configurations, and pre-installed libraries. +It provides an easy way for developers to get started working in a repository, +especially if they are uncomfortable
+Go to the "Code" dropdown from the main repository page, +select the three dot dropdown, and select "New with options..." +This will allow more configuration than the default codespace.
+
In the codespace configuration form, you will have an option to add "Recommended Secrets". +This is where you can add your personal Snowflake credentials to your codespace, +allowing for development against our Snowflake warehouse, including using dbt. +You should only add credentials for accounts that are protected by multi-factor authentication (MFA).
+
After you have added your secrets, click "Create Codespace". +Building it may take a few minutes, +but then you should be redirected to a VS Code environment in your browser.
+Once your codespace is created, you should be able to launch it +without re-creating it every time using the "Code" dropdown, +going to "Open in...", and selecting "Open in browser":
+
Once you have created and configured a codespace, +you have access to a relatively full-featured VS Code-based development environment. +This includes:
+--no-verify:
+ git commit --no-verify -m "Your commit message"
+The Data Services and Engineering team maintains a derived dataset from the +Microsoft US Building Footprints +and Global ML Building Footprints datasets. +The two datasets are broadly similar, but the latter has global coverage and is more frequently updated.
+We take the original datasets, and join them with US Census TIGER data to make them more useful +for demographic and social science research. The additional census-derived fields include:
+If a footprint intersects more than one of the above, +we assign it to the one with the greater intersection, +so each footprint should only appear once in the dataset.
+Note
+Despite the names, these derived datasets are scoped to California only.
+The data are stored as files in AWS S3. +We distribute them in both GeoParquet +and zipped Shapefile formats.
+GeoParquet is usually a superior format for doing data analytics as it is:
+However, GeoParquet is also somewhat newer, and not supported by all tooling yet, +so the zipped Shapefiles may be better suited for some workflows (especially Esri ones).
+GeoPandas +is an extension to the Python Pandas library enabling analysis of geopsatial vector data.
+Examples for reading the files using GeoPanas:
+import os
+import geopandas
+
+# Ensure S3 requests are anonymous, there is no need for AWS credentials here.
+os.environ["AWS_NO_SIGN_REQUEST"] = "YES"
+
+# Read GeoParquet using S3 URL
+gdf = geopandas.read_parquet(
+ "s3://dof-demographics-dev-us-west-2-public/"
+ "global_ml_building_footprints/parquet/county_fips_003.parquet"
+)
+
+# Read Shapefile using HTTPS URL
+gdf = geopandas.read_file(
+ "https://dof-demographics-dev-us-west-2-public.s3.us-west-2.amazonaws.com/"
+ "global_ml_building_footprints/shp/county_fips_003.zip"
+)
+Fennis Reed at the California Department of Finance Demographics Research Unit has created an ArcGIS Pro +toolbox +for downloading individual footprint files, +which can be downloaded here.
+The following tables contains public links to the datasets partitioned by county. +The HTTPS URLs can be used to directly download files using a web browser, +while the S3 URLs are more appropriate for scripts like the examples above.
+| County | +GeoParquet (HTTPS) | +Shapefile (HTTPS) | +GeoParquet (S3) | +Shapefile (S3) | +
|---|---|---|---|---|
| Alameda | +URL | +URL | +URL | +URL | +
| Alpine | +URL | +URL | +URL | +URL | +
| Amador | +URL | +URL | +URL | +URL | +
| Butte | +URL | +URL | +URL | +URL | +
| Calaveras | +URL | +URL | +URL | +URL | +
| Colusa | +URL | +URL | +URL | +URL | +
| Contra Costa | +URL | +URL | +URL | +URL | +
| Del Norte | +URL | +URL | +URL | +URL | +
| El Dorado | +URL | +URL | +URL | +URL | +
| Fresno | +URL | +URL | +URL | +URL | +
| Glenn | +URL | +URL | +URL | +URL | +
| Humboldt | +URL | +URL | +URL | +URL | +
| Imperial | +URL | +URL | +URL | +URL | +
| Inyo | +URL | +URL | +URL | +URL | +
| Kern | +URL | +URL | +URL | +URL | +
| Kings | +URL | +URL | +URL | +URL | +
| Lake | +URL | +URL | +URL | +URL | +
| Lassen | +URL | +URL | +URL | +URL | +
| Los Angeles | +URL | +URL | +URL | +URL | +
| Madera | +URL | +URL | +URL | +URL | +
| Marin | +URL | +URL | +URL | +URL | +
| Mariposa | +URL | +URL | +URL | +URL | +
| Mendocino | +URL | +URL | +URL | +URL | +
| Merced | +URL | +URL | +URL | +URL | +
| Modoc | +URL | +URL | +URL | +URL | +
| Mono | +URL | +URL | +URL | +URL | +
| Monterey | +URL | +URL | +URL | +URL | +
| Napa | +URL | +URL | +URL | +URL | +
| Nevada | +URL | +URL | +URL | +URL | +
| Orange | +URL | +URL | +URL | +URL | +
| Placer | +URL | +URL | +URL | +URL | +
| Plumas | +URL | +URL | +URL | +URL | +
| Riverside | +URL | +URL | +URL | +URL | +
| Sacramento | +URL | +URL | +URL | +URL | +
| San Benito | +URL | +URL | +URL | +URL | +
| San Bernardino | +URL | +URL | +URL | +URL | +
| San Diego | +URL | +URL | +URL | +URL | +
| San Francisco | +URL | +URL | +URL | +URL | +
| San Joaquin | +URL | +URL | +URL | +URL | +
| San Luis Obispo | +URL | +URL | +URL | +URL | +
| San Mateo | +URL | +URL | +URL | +URL | +
| Santa Barbara | +URL | +URL | +URL | +URL | +
| Santa Clara | +URL | +URL | +URL | +URL | +
| Santa Cruz | +URL | +URL | +URL | +URL | +
| Shasta | +URL | +URL | +URL | +URL | +
| Sierra | +URL | +URL | +URL | +URL | +
| Siskiyou | +URL | +URL | +URL | +URL | +
| Solano | +URL | +URL | +URL | +URL | +
| Sonoma | +URL | +URL | +URL | +URL | +
| Stanislaus | +URL | +URL | +URL | +URL | +
| Sutter | +URL | +URL | +URL | +URL | +
| Tehama | +URL | +URL | +URL | +URL | +
| Trinity | +URL | +URL | +URL | +URL | +
| Tulare | +URL | +URL | +URL | +URL | +
| Tuolumne | +URL | +URL | +URL | +URL | +
| Ventura | +URL | +URL | +URL | +URL | +
| Yolo | +URL | +URL | +URL | +URL | +
| Yuba | +URL | +URL | +URL | +URL | +
| County | +GeoParquet (HTTPS) | +Shapefile (HTTPS) | +GeoParquet (S3) | +Shapefile (S3) | +
|---|---|---|---|---|
| Alameda | +URL | +URL | +URL | +URL | +
| Alpine | +URL | +URL | +URL | +URL | +
| Amador | +URL | +URL | +URL | +URL | +
| Butte | +URL | +URL | +URL | +URL | +
| Calaveras | +URL | +URL | +URL | +URL | +
| Colusa | +URL | +URL | +URL | +URL | +
| Contra Costa | +URL | +URL | +URL | +URL | +
| Del Norte | +URL | +URL | +URL | +URL | +
| El Dorado | +URL | +URL | +URL | +URL | +
| Fresno | +URL | +URL | +URL | +URL | +
| Glenn | +URL | +URL | +URL | +URL | +
| Humboldt | +URL | +URL | +URL | +URL | +
| Imperial | +URL | +URL | +URL | +URL | +
| Inyo | +URL | +URL | +URL | +URL | +
| Kern | +URL | +URL | +URL | +URL | +
| Kings | +URL | +URL | +URL | +URL | +
| Lake | +URL | +URL | +URL | +URL | +
| Lassen | +URL | +URL | +URL | +URL | +
| Los Angeles | +URL | +URL | +URL | +URL | +
| Madera | +URL | +URL | +URL | +URL | +
| Marin | +URL | +URL | +URL | +URL | +
| Mariposa | +URL | +URL | +URL | +URL | +
| Mendocino | +URL | +URL | +URL | +URL | +
| Merced | +URL | +URL | +URL | +URL | +
| Modoc | +URL | +URL | +URL | +URL | +
| Mono | +URL | +URL | +URL | +URL | +
| Monterey | +URL | +URL | +URL | +URL | +
| Napa | +URL | +URL | +URL | +URL | +
| Nevada | +URL | +URL | +URL | +URL | +
| Orange | +URL | +URL | +URL | +URL | +
| Placer | +URL | +URL | +URL | +URL | +
| Plumas | +URL | +URL | +URL | +URL | +
| Riverside | +URL | +URL | +URL | +URL | +
| Sacramento | +URL | +URL | +URL | +URL | +
| San Benito | +URL | +URL | +URL | +URL | +
| San Bernardino | +URL | +URL | +URL | +URL | +
| San Diego | +URL | +URL | +URL | +URL | +
| San Francisco | +URL | +URL | +URL | +URL | +
| San Joaquin | +URL | +URL | +URL | +URL | +
| San Luis Obispo | +URL | +URL | +URL | +URL | +
| San Mateo | +URL | +URL | +URL | +URL | +
| Santa Barbara | +URL | +URL | +URL | +URL | +
| Santa Clara | +URL | +URL | +URL | +URL | +
| Santa Cruz | +URL | +URL | +URL | +URL | +
| Shasta | +URL | +URL | +URL | +URL | +
| Sierra | +URL | +URL | +URL | +URL | +
| Siskiyou | +URL | +URL | +URL | +URL | +
| Solano | +URL | +URL | +URL | +URL | +
| Sonoma | +URL | +URL | +URL | +URL | +
| Stanislaus | +URL | +URL | +URL | +URL | +
| Sutter | +URL | +URL | +URL | +URL | +
| Tehama | +URL | +URL | +URL | +URL | +
| Trinity | +URL | +URL | +URL | +URL | +
| Tulare | +URL | +URL | +URL | +URL | +
| Tuolumne | +URL | +URL | +URL | +URL | +
| Ventura | +URL | +URL | +URL | +URL | +
| Yolo | +URL | +URL | +URL | +URL | +
| Yuba | +URL | +URL | +URL | +URL | +
In most settings, what is considered acceptable performance is relative to business needs and constraints. It's not atypical to deem the performance acceptable as long as there are no scheduling conflicts and models can run within a timeframe dictated by the frequency of the models running. In other words, if you need to run models every hour then the entire job cannot take longer than an hour to run. In general compute costs are not so high to necessarily be worth optimizing the underlying queries but may be high enough to optimize frequency or data size.
+Although compute time is relatively cheap, it's sometimes possible with larger datasets that need to be frequently refreshed to optimize performance to save enough costs to be worth the time to optimize. In Snowflake you can easily monitor costs in the Admin/Usage section of the Snowflake UI, where you can see credits used by warehouse and role. +Snowflake also provides several tables with meta information that can be used to derive exact costs for each query - an approach to this, with a ready-use-query can be found in the Select.dev blog post "Calculating cost per query in Snowflake"
+Typically, unless model performance is obviously very poor you are better off adjusting the frequency of runs (end users almost always over-state their desire for data freshness) or reducing data set size either by limiting what you provide or by using incremental models.
+In other words, very often the questions you should be asking are not in the category of SQL performance tuning but rather "do we need this data to be this fresh?" and "do we need all this data?".
+Often performance issues show up in scheduling. If you are running jobs once a day it is extremely unlikely you will run into any scheduling conflicts. However, if a much higher frequency is required, it's possible for jobs to take longer than that time between runs. In this case a common first approach is to break up model runs so that things that don't need to run as frequently can run separately from models that require more frequent updating. A typical way of doing this is to either use dbt run --select or dbt tags dbt tags to select models in groups. +This is not to say performance tuning of individual queries is never worth it but that the big macro gains come more from running models less frequently and/or with less data, e.g. using filtering or incremental models.
+It is extremely important to balance time spent in optimizing model performance with compute costs and other concerns. If it takes you a day to optimize a model to run only a few seconds faster and save a few pennies per run, it's not likely worth the effort. Similarly, the use of incremental materilization can certainly reduce build time but introduce complexity and require a degree of monitoring to ensure integrity. See also Materialization Matters below.
+With every dbt run or build several artifacts are generated in the target/ directory, including the run_results.json file. This includes detailed information on run execution and many people parse this to create dashboards to report on dbt performance and help with optimization and cost monitoring. There is an important caveat here: simply knowing how long a model took to run is important to uncover which models might need optimization, but cannot tell you anything about why they are performing poorly.
+Every time you run a model dbt outputs timing, information which you can easily use identify non-performance models. The output will look like: +
14:16:39.438935 [info ] [Thread-4 ]: 136 of 160 OK created sql table model dbt_aerishan.JobControl_Published_CertEligActions [SUCCESS 1 in 43.00s]
+dbt Cloud has a nicer interface for finding which models in a project are running longest. Visit the Deploy > Runs section of dbt Cloud. You'll see a full list of jobs and how long each one toook. To drill down to the model timing level click on a run name. You can expand the "Invoke dbt build" section under "Run Summary" to get a detailed summary of your run as well as timing for each model and test. There is also a "Debug logs" section for even more detail, including the exact queries run and an option to download the logs for easier viewing. Of course this is also where you go to find model and test errors and warnings!
+
For a quick visual reference of which models take up the most time in a run, click on the "Model Timing" tab. If you hover over a model you will be shown the specific timing.
+
Snowflake has quite a lot of performance data readily available through it's information_schema.QUERY_HISTORY() table function and several views in the Account Usage schema. This is great not only for finding expensive queries regardless of source and of course for all sorts of analytics on Snowflake usage, such as credits.
The Query History gives you real time data while the Account Usage is delayed. So Query History is great for analyzing your own queries in development and for current query performance in production.
+Example Query: Get top time-consuming queries for the dbt Cloud production loads +
SELECT
+ query_text, query_type, database_name, schema_name,
+ user_name, total_elapsed_time
+ FROM
+ -- query_history() is a table function
+ table (information_schema.query_history())
+ WHERE user_name = 'DBT_CLOUD_SVC_USER_PRD'
+ORDER BY total_elapsed_time DESC
+LIMIT 20
+As you might have guessed this also lets you search for a model on query text, so you can find specific dbt models or classes of models: +
WHERE query_text LIKE '%stg_%'
+The Account Usage schema (snowflake.account_usage) has multiple views that are of interest for monitoring not just query performance and credit usage but warehouse and database usage and more. This data is delayed 45 minutes but has a much longer history.
+Example Query: Find the queries with highest total execution time this month for the dbt cloud production loads. +
SELECT query_text,
+ SUM(execution_time) / 60000 AS total_exec_time_mins,
+ SUM(credits_used_cloud_services) AS total_credits_used
+from snowflake.account_usage.query_history
+WHERE
+ start_time >= date_trunc(month, current_date)
+ AND user_name = 'DBT_CLOUD_SVC_USER_PRD'
+GROUP BY 1
+ORDER BY total_exec_time_mins DESC
+LIMIT 20
+Example Query: Get Credits used by Warehouse this month +
select warehouse_name,
+ sum(credits_used) as total_credits_used
+from warehouse_metering_history
+where start_time >= date_trunc(month, current_date)
+group by 1
+order by 2 desc;
+Now that you've identified which models might need optimization, it's time to figure out how to get them to run faster. These options are roughly in order of bang-for-buck in most situations.
+It's common for end-users to say they want the freshest data (who doesn't?) but in practice require a much lower frequency of refreshing. To gain an understand of the real-world needs it's helpful to see the frequency with which end-users actually view reporting and to consider the time scales involved. If someone only cares about monthly results, for example, you can in theory have a 30 day frequency for model runs. +It's also quite common to have parts of the data be relatively static, and only need to be refreshed occasionally whereas other parts of the data might change much more often. +An easy way to break up model runs is by using dbt tags. +
models:
+ foo_model:
+ +tags:
+ - "daily"
+ - "staging"
+ bar_model:
+ +tags: "hourly"
+ baz_)_model:
+ +tags: "weekly"
+In this case you can run certain models with:
+dbt run --select tag:daily
+Of course this works in dbt Cloud as well!
For more information refer to the dbt tags documentation.
+For a good comparison of materialization options and their trade-offs see the Materialization Best Practices section of the dbt docs.
+Views: Are a trade-off between build performance and read/reporting performance. In cases where you are using a BI tool, you should almost always use table materializations unless data storage size is an issue or refresh frequency is so high that cost or scheduling conflicts become a problem. In cases where performance at time of reporting is not an issue (say, you are generating an aggregated report on a monthly basis) then views can be a great way to cut run time. Another case where views can be a good option is with staging data of relatively small size, where your queries are relatively light-weight and you want to ensure fresh data without having to configure separate runs for those models.
+Incremental Models: +For a very large data sets, it can be essential to use incremental models. For this to work, you need some means of filtering records from the source table, typically using a timestamp. You then add a conditional block into your model to only select new records unless you're doing a full refresh. It's worth noting that incremental models can be tricky to get right and you will often want to implement some additional data integrity testing to ensure data is fresh and complete. For a more detailed discussion of Incremental Models, see Incremental models in-depth
+An example in our current projects is the CalHR Ecos model stg_CertEligibles. This query takes over three minutes to run and no wonder - it generates 858 million rows! This is clearly a case where we should ask if we need all of that data or can filtered in someway and if the answer is yes, then we should consider using an incremental materialization.
+A great many books have been written on this subject! The good news is that most of the tools we use provide excellent resources for analyzing query performance.
+Because models are often created using a CREATE TABLE... SELECT statement you need to separate out read from write performance to understand if the issue is that your original query is slow or you are simply moving a lot of data and it takes time. It's worth saying that the chances are good that if you are moving a lot of data you are also querying a lot of data and in fact both read and write may be very time consuming but this is not a given -- if you are doing lots of joins on big data sets along with aggregations that output a small number of rows, then probably your model performance is read-bound. If this is the case the first question you should probably ask is can you break up that model into smaller chunks using staging and intermediate models.
+A good way to get a sense of read vs write performance is to do one or more of: +1. Simply know the number of rows generated by the model (for some database dbt will output this in the output above). If you are creating tables with millions of rows you should probably consider an incremental model or reassess if you can filter and narrow your data somehow. +2. Use your database's query profiler, if available, to separate out what part of the execution is taking the most amount of time. In Snowflake for example, you can use the query profile to easily determine whether a query is rebound or write down and also determine where exactly other performance issues may lie. A CREATE TABLE with a simple select, for example, will show that the majority of time is spent in the CreateTableAsSelect node and only a fraction of the time in the Result node. +Be careful if you are comparing queries across runs - most databases use caching and this will of course affect your results (see Caching Notes below). +3. Switch the materialization to view. Typically a view will take a fraction of the time to generate, and if that's the case you know your model is slow in writes. +4. Run the query separately in the database without the CREATE TABLE part. When you do this you can typically assess the execution plan
+You can easily pull up the query profile for any query that has been run in Snowflake either from a worksheet or from the query history page. This includes queries run from dbt Cloud! This profile is essential in understanding the elements of your query that are most costly in terms of time, and which might be improved through optimization. Refer to the Analyzing Queries Using Query Profile page in the Snowflake Documentation for complete information including common problems and their solutions.
+Big Query offers similar query execution profiling in the Google Cloud Console. See Query plan and timeline as well as Big Query's Introduction to optimizing query performance
+Most databases use some type of caching which needs to be turned off in order to properly test performance. Snowflake uses both a results cache and a disk cache, but only one can be turned off with a session variable: +
alter session set use_cached_result = false;
+{% if target.name = 'dev' %}
+ LIMIT 500
+{% endif %}Models in a data warehouse do not follow the same naming conventions as raw cloud resources, +as their most frequent consumers are analytics engineers and data analysts.
+The following conventions are used where appropriate:
+Dimension tables are prefixed with dim_.
Fact tables are prefixed with fct_.
Staging tables are prefixed with stg_.
Intermediate tables are prefixed with int_.
We may adopt additional conventions for denoting aggregations, column data types, etc. in the future. +If during the course of a project's model development we determine that simpler human-readable names +work better for our partners or downstream consumers, we may drop the above prefixing conventions.
+We broadly follow the architecture described in +this dbt blog post +for our Snowflake dbt project.
+It is described in more detail in our Snowflake docs.
+dbt's default method for generating custom schema names +works well for a single-database setup:
+A downside of the default is that production models all get a prefix, +which may not be an ideal naming convention for end-users.
+Because our architecture separates development and production databases,
+and has strict permissions protecting the RAW database,
+there is less danger of breaking production models.
+So we use our own custom schema name following the modified from the
+approach of the GitLab Data Team.
In production, each schema is just the custom schema name without any prefix.
+In non-production environments the default is used, where analytics engineers
+get the custom schema name prefixed with their target schema name (i.e. dbt_username_schemaname),
+and CI runs get the custom schema name prefixed with a CI job name.
This approach may be reevaluated as the project matures.
+Our Snowflake architecture allows for reasonably safe SELECTing from the production RAW database while developing models.
+While this could be expensive for large tables,
+it also allows for faster and more reliable model development.
To develop against production RAW data, first you need someone with the USERADMIN role to grant rights to the TRANSFORMER_DEV role
+(this need only be done once, and can be revoked later):
USE ROLE USERADMIN;
+GRANT ROLE RAW_PRD_READ TO ROLE TRANSFORMER_DEV;
+Note
+This grant is not managed via terraform in order to keep the configurations of +different environments as logically separate as possible. We may revisit this +decision should the manual grant cause problems.
+You can then run dbt locally and specify the RAW database manually:
DBT_RAW_DB=RAW_PRD dbt run
+82`IkS(P
z*+J-bX20u}X_m|SHuYlP@AN(^4}x*DTy(E-xLf*;bYHKz+}B}utAx;LWHC`LOQ}`k
zvk#7iR_Ps9tTWD@&t7=&-1$e$WUSR;vCi%xLFd