Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Private packages beta #6675

Merged
merged 20 commits into from
Dec 17, 2024
Merged
Changes from 1 commit
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
f90ef93
Private packages beta
matthewshaver Dec 16, 2024
3b89966
Update website/docs/docs/build/packages.md
matthewshaver Dec 16, 2024
ed10f19
Update website/docs/docs/build/packages.md
matthewshaver Dec 16, 2024
fb75507
Merge branch 'current' into private-packages
mirnawong1 Dec 17, 2024
80a45d3
Update website/docs/docs/build/packages.md
mirnawong1 Dec 17, 2024
71b4813
Update website/docs/docs/build/packages.md
mirnawong1 Dec 17, 2024
efe5e05
Update website/docs/docs/build/packages.md
mirnawong1 Dec 17, 2024
c17f144
Update website/docs/docs/build/packages.md
mirnawong1 Dec 17, 2024
ed94b87
Update website/docs/docs/build/packages.md
matthewshaver Dec 17, 2024
ccda55d
Update website/docs/docs/build/packages.md
matthewshaver Dec 17, 2024
a034bbd
Update website/docs/docs/build/packages.md
matthewshaver Dec 17, 2024
d1fc437
Update website/docs/docs/build/packages.md
matthewshaver Dec 17, 2024
b34d197
Fixing code spacing
matthewshaver Dec 17, 2024
0e4e05e
Update website/docs/docs/build/packages.md
matthewshaver Dec 17, 2024
e394eb0
Update website/docs/docs/build/packages.md
matthewshaver Dec 17, 2024
ef3e6e7
Update website/docs/docs/build/packages.md
matthewshaver Dec 17, 2024
121436b
Update website/docs/docs/build/packages.md
matthewshaver Dec 17, 2024
c622069
Update website/docs/docs/build/packages.md
matthewshaver Dec 17, 2024
87dcc54
Merge branch 'current' into private-packages
matthewshaver Dec 17, 2024
342d913
Update website/docs/docs/build/packages.md
matthewshaver Dec 17, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
70 changes: 66 additions & 4 deletions website/docs/docs/build/packages.md
Original file line number Diff line number Diff line change
Expand Up @@ -157,9 +157,64 @@ packages:

Where `name: 'dbt_utils'` specifies the subfolder of `dbt_packages` that's created for the package source code to be installed within.

### Private packages
## Private packages

matthewshaver marked this conversation as resolved.
Show resolved Hide resolved
#### SSH Key Method (Command Line only)
### Native private packages <Lifecycle status='beta'/>

dbt Cloud supports private packages from [supported](#prerequisites) Git repos leveraging an exisiting [configuration](/docs/cloud/git/git-configuration-in-dbt-cloud) in your environment. Previously, you had to configure a [token](#git-token-method) to retrieve packages from your private repos.

#### Prerequisites

Native support for private packages is supported on the following services:
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Contributor

@mirnawong1 mirnawong1 Dec 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for this whole 'prerequisites' section, might be helpful to organize it all in bullets and link to the account integrations page. like this:

#### Prerequisites

- The following providers natively support private packages:
  - [GitHub](https://docs-getdbt-com-git-private-packages-dbt-labs.vercel.app/docs/cloud/git/connect-github)
  - [Azure DevOps](https://docs-getdbt-com-git-private-packages-dbt-labs.vercel.app/docs/cloud/git/connect-azure-devops)
  - GitLab support is coming soon.

- You need a supported Git provider configured in the [**Integrations** section](/docs/cloud/account-integrations) of your **Account settings** to use this feature.

- [Github](/docs/cloud/git/connect-github)
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved
- [Azure DevOps](/docs/cloud/git/connect-azure-devops)

matthewshaver marked this conversation as resolved.
Show resolved Hide resolved
Support for Gitlab is coming soon.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if you accept above, remember to remove this

matthewshaver marked this conversation as resolved.
Show resolved Hide resolved

You must have the Git service configured in the **Integrations** section of your **Account settings** to use this feature.
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved

#### Configuration

Use the `private` block in your `packages.yml` or `dependencies.yml` to clone package repos using your existing dbt Cloud Git integration without having to provision an access token or create a dbt Cloud environment variable:
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved

<File name="packages.yml">

```yaml
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved

mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved
packages:
- private: dbt-labs/awesome_repo

- package: normal packages

[...]

mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved
```

</File>

You can pin private packages similar to regular dbt packages:
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved

```yaml

mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved
packages:
- private: dbt-labs/awesome_repo
Copy link
Contributor

@mirnawong1 mirnawong1 Dec 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as above comment about yaml indent

revision: branch name | tagged git release | a specific commit (full 40-character hash}

```

If you are using multiple Git integrations, disambiguate by adding the provider key:
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved

```yaml

mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved
packages:
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved
- private: dbt-labs/awesome_repo
provider: "github" | "azure"

```

With this method, you can retrieve private packages from an integrated Git provider without any additional steps to connect.

### SSH Key Method (Command Line only)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] reported by reviewdog 🐶
[custom.SentenceCaseHeaders] 'SSH Key Method (Command Line only)' should use sentence-style capitalization. Try 'SSH key method command line only' instead.

matthewshaver marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] reported by reviewdog 🐶
[custom.SentenceCaseHeaders] 'SSH Key Method (Command Line only)' should use sentence-style capitalization. Try 'SSH key method command line only' instead.

If you're using the Command Line, private packages can be cloned via SSH and an SSH key.

When you use SSH keys to authenticate to your git remote server, you don’t need to supply your username and password each time. Read more about SSH keys, how to generate them, and how to add them to your git provider here: [Github](https://docs.github.com/en/github/authenticating-to-github/connecting-to-github-with-ssh) and [GitLab](https://docs.gitlab.com/ee/user/ssh.html).
Expand All @@ -177,7 +232,14 @@ packages:
If you're using dbt Cloud, the SSH key method will not work, but you can use the [HTTPS Git Token Method](https://docs.getdbt.com/docs/build/packages#git-token-method).


#### Git token method
### Git token method

:::note

dbt Cloud has [native support](#native-private-packages) for Git hosted private packages for Github and Azure DevOps (GitLab coming soon). If you are using a supported [integrated Git environment](/docs/cloud/git/git-configuration-in-dbt-cloud), you no longer need to configure Git tokens to retrieve private packages.
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved

:::

This method allows the user to clone via HTTPS by passing in a git token via an environment variable. Be careful of the expiration date of any token you use, as an expired token could cause a scheduled run to fail. Additionally, user tokens can create a challenge if the user ever loses access to a specific repo.


Expand Down Expand Up @@ -246,7 +308,7 @@ Read more about creating a Personal Access Token [here](https://confluence.atlas



#### Configure subdirectory for packaged projects
## Configure subdirectory for packaged projects

In general, dbt expects `dbt_project.yml` to be located as a top-level file in a package. If the packaged project is instead nested in a subdirectory—perhaps within a much larger mono repo—you can optionally specify the folder path as `subdirectory`. dbt will attempt a [sparse checkout](https://git-scm.com/docs/git-sparse-checkout) of just the files located within that subdirectory. Note that you must be using a recent version of `git` (`>=2.26.0`).

Expand Down
Loading