Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Private packages beta #6675

Merged
merged 20 commits into from
Dec 17, 2024
Merged
Changes from 14 commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
f90ef93
Private packages beta
matthewshaver Dec 16, 2024
3b89966
Update website/docs/docs/build/packages.md
matthewshaver Dec 16, 2024
ed10f19
Update website/docs/docs/build/packages.md
matthewshaver Dec 16, 2024
fb75507
Merge branch 'current' into private-packages
mirnawong1 Dec 17, 2024
80a45d3
Update website/docs/docs/build/packages.md
mirnawong1 Dec 17, 2024
71b4813
Update website/docs/docs/build/packages.md
mirnawong1 Dec 17, 2024
efe5e05
Update website/docs/docs/build/packages.md
mirnawong1 Dec 17, 2024
c17f144
Update website/docs/docs/build/packages.md
mirnawong1 Dec 17, 2024
ed94b87
Update website/docs/docs/build/packages.md
matthewshaver Dec 17, 2024
ccda55d
Update website/docs/docs/build/packages.md
matthewshaver Dec 17, 2024
a034bbd
Update website/docs/docs/build/packages.md
matthewshaver Dec 17, 2024
d1fc437
Update website/docs/docs/build/packages.md
matthewshaver Dec 17, 2024
b34d197
Fixing code spacing
matthewshaver Dec 17, 2024
0e4e05e
Update website/docs/docs/build/packages.md
matthewshaver Dec 17, 2024
e394eb0
Update website/docs/docs/build/packages.md
matthewshaver Dec 17, 2024
ef3e6e7
Update website/docs/docs/build/packages.md
matthewshaver Dec 17, 2024
121436b
Update website/docs/docs/build/packages.md
matthewshaver Dec 17, 2024
c622069
Update website/docs/docs/build/packages.md
matthewshaver Dec 17, 2024
87dcc54
Merge branch 'current' into private-packages
matthewshaver Dec 17, 2024
342d913
Update website/docs/docs/build/packages.md
matthewshaver Dec 17, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
64 changes: 60 additions & 4 deletions website/docs/docs/build/packages.md
Original file line number Diff line number Diff line change
Expand Up @@ -157,9 +157,58 @@ packages:

Where `name: 'dbt_utils'` specifies the subfolder of `dbt_packages` that's created for the package source code to be installed within.

### Private packages
## Private packages

matthewshaver marked this conversation as resolved.
Show resolved Hide resolved
#### SSH Key Method (Command Line only)
### Native private packages <Lifecycle status='beta'/>

dbt Cloud supports private packages from [supported](#prerequisites) Git repos leveraging an exisiting [configuration](/docs/cloud/git/git-configuration-in-dbt-cloud) in your environment. Previously, you had to configure a [token](#git-token-method) to retrieve packages from your private repos.

#### Prerequisites

dbt Cloud natively supports private packages for the following services:
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved
- [GitHub](/docs/cloud/git/connect-github)
- [Azure DevOps](/docs/cloud/git/connect-azure-devops)
- Support for GitLab is coming soon.

To use this feature, you must have a supported Git provider configured in the **Integrations** section of your **Account settings**.
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved

#### Configuration

Use the `private` block in your `packages.yml` or `dependencies.yml` to clone package repos using your existing dbt Cloud Git integration without having to provision an access token or create a dbt Cloud environment variable:
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved

<File name="packages.yml">

```yaml
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved
packages:
- private: dbt-labs/awesome_repo
- package: normal packages

mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved
[...]
```

</File>

You can pin private packages similar to regular dbt packages:
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved

```yaml
packages:
- private: dbt-labs/awesome_repo
revision: "0.9.5" # Pin to a tag, branch, or complete 40-character commit hash

```

If you are using multiple Git integrations, disambiguate by adding the provider key:
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved

```yaml
packages:
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved
- private: dbt-labs/awesome_repo
provider: "github" | "azure"
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved

```

With this method, you can retrieve private packages from an integrated Git provider without any additional steps to connect.

### SSH key method (command line only)
If you're using the Command Line, private packages can be cloned via SSH and an SSH key.

When you use SSH keys to authenticate to your git remote server, you don’t need to supply your username and password each time. Read more about SSH keys, how to generate them, and how to add them to your git provider here: [Github](https://docs.github.com/en/github/authenticating-to-github/connecting-to-github-with-ssh) and [GitLab](https://docs.gitlab.com/ee/user/ssh.html).
Expand All @@ -177,7 +226,14 @@ packages:
If you're using dbt Cloud, the SSH key method will not work, but you can use the [HTTPS Git Token Method](https://docs.getdbt.com/docs/build/packages#git-token-method).


#### Git token method
### Git token method

:::note

dbt Cloud has [native support](#native-private-packages) for Git hosted private packages for Github and Azure DevOps (GitLab coming soon). If you are using a supported [integrated Git environment](/docs/cloud/git/git-configuration-in-dbt-cloud), you no longer need to configure Git tokens to retrieve private packages.
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved

:::

This method allows the user to clone via HTTPS by passing in a git token via an environment variable. Be careful of the expiration date of any token you use, as an expired token could cause a scheduled run to fail. Additionally, user tokens can create a challenge if the user ever loses access to a specific repo.


Expand Down Expand Up @@ -246,7 +302,7 @@ Read more about creating a Personal Access Token [here](https://confluence.atlas



#### Configure subdirectory for packaged projects
## Configure subdirectory for packaged projects

In general, dbt expects `dbt_project.yml` to be located as a top-level file in a package. If the packaged project is instead nested in a subdirectory—perhaps within a much larger mono repo—you can optionally specify the folder path as `subdirectory`. dbt will attempt a [sparse checkout](https://git-scm.com/docs/git-sparse-checkout) of just the files located within that subdirectory. Note that you must be using a recent version of `git` (`>=2.26.0`).

Expand Down
Loading