Skip to content

Commit

Permalink
Docs: capitalize TOML (#1909)
Browse files Browse the repository at this point in the history
  • Loading branch information
burnash authored Oct 2, 2024
1 parent df54789 commit 543044e
Show file tree
Hide file tree
Showing 17 changed files with 34 additions and 30 deletions.
2 changes: 1 addition & 1 deletion docs/website/docs/dlt-ecosystem/destinations/athena.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ so pip does not fail on backtracking.

To edit the `dlt` credentials file with your secret info, open `.dlt/secrets.toml`. You will need to provide a `bucket_url`, which holds the uploaded parquet files, a `query_result_bucket`, which Athena uses to write query results to, and credentials that have write and read access to these two buckets as well as the full Athena access AWS role.

The toml file looks like this:
The TOML file looks like this:

```toml
[destination.filesystem]
Expand Down
2 changes: 1 addition & 1 deletion docs/website/docs/dlt-ecosystem/destinations/clickhouse.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,7 @@ To load data into ClickHouse, you need to create a ClickHouse database. While we
2. You can pass a database connection string similar to the one used by the `clickhouse-driver` library. The credentials above will look like this:

```toml
# Keep it at the top of your toml file before any section starts.
# Keep it at the top of your TOML file, before any section starts
destination.clickhouse.credentials="clickhouse://dlt:Dlt*12345789234567@localhost:9000/dlt?secure=1"
```

Expand Down
2 changes: 1 addition & 1 deletion docs/website/docs/dlt-ecosystem/destinations/dremio.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ or with `pip install "dlt[dremio,s3]"` which will install `s3fs`, `pyarrow`, and

To edit the `dlt` credentials file with your secret info, open `.dlt/secrets.toml`. You will need to provide a `bucket_url` which holds the uploaded parquet files.

The toml file looks like this:
The TOML file looks like this:

```toml
[destination.filesystem]
Expand Down
2 changes: 1 addition & 1 deletion docs/website/docs/dlt-ecosystem/destinations/filesystem.md
Original file line number Diff line number Diff line change
Expand Up @@ -234,7 +234,7 @@ export DESTINATION__FILESYSTEM__KWARGS = '{"auto_mkdir": true/false}'
bucket_url = 'C:\a\b\c'
```

In the example above, we specify `bucket_url` using **toml's literal strings** that do not require [escaping of backslashes](https://github.com/toml-lang/toml/blob/main/toml.md#string).
In the example above, we specify `bucket_url` using **TOML's literal strings** that do not require [escaping of backslashes](https://github.com/toml-lang/toml/blob/main/toml.md#string).

```toml
[destination.unc_destination]
Expand Down
4 changes: 2 additions & 2 deletions docs/website/docs/dlt-ecosystem/destinations/mssql.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ LongAsMax="yes"

You can also pass a SQLAlchemy-like database connection:
```toml
# keep it at the top of your toml file! before any section starts
# Keep it at the top of your TOML file, before any section starts
destination.mssql.credentials="mssql://loader:<password>@loader.database.windows.net/dlt_data?TrustServerCertificate=yes&Encrypt=yes&LongAsMax=yes"
```

Expand Down Expand Up @@ -145,7 +145,7 @@ driver="ODBC Driver 18 for SQL Server"
When using a SQLAlchemy connection string, replace spaces with `+`:

```toml
# keep it at the top of your toml file! before any section starts
# Keep it at the top of your TOML file, before any section starts
destination.mssql.credentials="mssql://loader:<password>@loader.database.windows.net/dlt_data?driver=ODBC+Driver+18+for+SQL+Server"
```

Expand Down
6 changes: 3 additions & 3 deletions docs/website/docs/dlt-ecosystem/destinations/postgres.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ connect_timeout = 15

You can also pass a database connection string similar to the one used by the `psycopg2` library or [SQLAlchemy](https://docs.sqlalchemy.org/en/20/core/engines.html#postgresql). The credentials above will look like this:
```toml
# keep it at the top of your toml file! before any section starts
# Keep it at the top of your TOML file, before any section starts
destination.postgres.credentials="postgresql://loader:<password>@localhost/dlt_data?connect_timeout=15"
```

Expand All @@ -86,11 +86,11 @@ If you set the [`replace` strategy](../../general-usage/full-loading.md) to `sta
`postgres` supports various timestamp types, which can be configured using the column flags `timezone` and `precision` in the `dlt.resource` decorator or the `pipeline.run` method.

- **Precision**: allows you to specify the number of decimal places for fractional seconds, ranging from 0 to 6. It can be used in combination with the `timezone` flag.
- **Timezone**:
- **Timezone**:
- Setting `timezone=False` maps to `TIMESTAMP WITHOUT TIME ZONE`.
- Setting `timezone=True` (or omitting the flag, which defaults to `True`) maps to `TIMESTAMP WITH TIME ZONE`.

#### Example precision and timezone: TIMESTAMP (3) WITHOUT TIME ZONE
#### Example precision and timezone: TIMESTAMP (3) WITHOUT TIME ZONE
```py
@dlt.resource(
columns={"event_tstamp": {"data_type": "timestamp", "precision": 3, "timezone": False}},
Expand Down
2 changes: 1 addition & 1 deletion docs/website/docs/dlt-ecosystem/destinations/redshift.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ To load data into Redshift, you need to create a Redshift cluster and enable acc

You can also pass a database connection string similar to the one used by the `psycopg2` library or [SQLAlchemy](https://docs.sqlalchemy.org/en/20/core/engines.html#postgresql). The credentials above will look like this:
```toml
# Keep it at the top of your toml file! Before any section starts
# Keep it at the top of your TOML file, before any section starts
destination.redshift.credentials="redshift://loader:<password>@localhost/dlt_data?connect_timeout=15"
```

Expand Down
8 changes: 5 additions & 3 deletions docs/website/docs/dlt-ecosystem/destinations/snowflake.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ The **password authentication** is not any different from other databases like P

You can also pass credentials as a database connection string. For example:
```toml
# keep it at the top of your toml file! before any section starts
# Keep it at the top of your TOML file, before any section starts
destination.snowflake.credentials="snowflake://loader:<password>@kgiotue-wn98412/dlt_data?warehouse=COMPUTE_WH&role=DLT_LOADER_ROLE"

```
Expand All @@ -98,7 +98,7 @@ private_key_passphrase="passphrase"
If you pass a passphrase in the connection string, please URL encode it.
```toml
# keep it at the top of your toml file! before any section starts
# Keep it at the top of your TOML file, before any section starts
destination.snowflake.credentials="snowflake://loader:<password>@kgiotue-wn98412/dlt_data?private_key=<base64 encoded pem>&private_key_passphrase=<url encoded passphrase>"
```

Expand All @@ -117,6 +117,7 @@ In the case of external authentication, you need to find documentation for your
### Additional connection options

We pass all query parameters to the `connect` function of the Snowflake Python Connector. For example:

```toml
[destination.snowflake.credentials]
database = "dlt_data"
Expand All @@ -126,7 +127,8 @@ timezone="UTC"
# keep session alive beyond 4 hours
client_session_keep_alive=true
```
This will set the timezone and session keep alive. Mind that if you use `toml`, your configuration is typed. The alternative:

This will set the timezone and session keep alive. Mind that if you use TOML, your configuration is typed. The alternative:
`"snowflake://loader/dlt_data?authenticator=oauth&timezone=UTC&client_session_keep_alive=true"`
will pass `client_session_keep_alive` as a string to the connect method (which we didn't verify if it works).

Expand Down
4 changes: 2 additions & 2 deletions docs/website/docs/dlt-ecosystem/destinations/synapse.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,7 @@ host = "your_synapse_workspace_name.sql.azuresynapse.net"
Equivalently, you can also pass a connection string as follows:

```toml
# keep it at the top of your toml file! before any section starts
# Keep it at the top of your TOML file, before any section starts
destination.synapse.credentials = "synapse://loader:your_loader_password@your_synapse_workspace_name.azuresynapse.net/yourpool"
```

Expand Down Expand Up @@ -215,7 +215,7 @@ connect_timeout = 15
`port` and `connect_timeout` can also be included in the connection string:

```toml
# keep it at the top of your toml file! before any section starts
# Keep it at the top of your TOML file, before any section starts
destination.synapse.credentials = "synapse://loader:your_loader_password@your_synapse_workspace_name.azuresynapse.net:1433/yourpool?connect_timeout=15"
```

Expand Down
2 changes: 1 addition & 1 deletion docs/website/docs/dlt-ecosystem/staging.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ In essence, you need to set up two destinations and then pass them to `dlt.pipel

Please follow our guide in the [redshift destination documentation](destinations/redshift.md). In your `secrets.toml`, you added:
```toml
# keep it at the top of your toml file! before any section starts
# Keep it at the top of your TOML file, before any section starts
destination.redshift.credentials="redshift://loader:<password>@localhost/dlt_data?connect_timeout=15"
```

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -253,7 +253,7 @@ bucket_url='~\Documents\csv_files\'
</Tabs>

You can also specify the credentials using environment variables. The name of the corresponding environment
variable should be slightly different from the corresponding name in the `toml` file. Simply replace dots `.` with double
variable should be slightly different from the corresponding name in the TOML file. Simply replace dots `.` with double
underscores `__`:

```sh
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -134,13 +134,12 @@ source = sql_database(
dlt.pipeline("demo").run(source)
```

## Configuring with TOML/environment variables

You can set most of the arguments of `sql_database()` and `sql_table()` directly in the `.toml` files and/or as environment variables. `dlt` automatically injects these values into the pipeline script.
## Configuring with TOML or environment variables
You can set most of the arguments of `sql_database()` and `sql_table()` directly in the TOML files or as environment variables. `dlt` automatically injects these values into the pipeline script.

This is particularly useful with `sql_table()` because you can maintain a separate configuration for each table (below we show **secrets.toml** and **config.toml**; you are free to combine them into one):

The examples below show how you can set arguments in any of the `.toml` files (`secrets.toml` or `config.toml`):
The examples below show how you can set arguments in any of the TOML files (`secrets.toml` or `config.toml`):
1. Specifying connection string:
```toml
[sources.sql_database]
Expand Down
6 changes: 3 additions & 3 deletions docs/website/docs/general-usage/credentials/advanced.md
Original file line number Diff line number Diff line change
Expand Up @@ -91,9 +91,9 @@ Now,
* connection string (used in SQL Alchemy) (in code and via config providers).
* if nothing is passed, the default credentials are used (i.e., those present on Cloud Function runner)

## Toml files structure
## TOML files structure

`dlt` arranges the sections of [toml files](setup/#secretstoml-and-configtoml) into a **default layout** that is expected by the [injection mechanism](#injection-mechanism).
`dlt` arranges the sections of [TOML files](setup/#secretstoml-and-configtoml) into a **default layout** that is expected by the [injection mechanism](#injection-mechanism).
This layout makes it easy to configure simple cases but also provides room for more explicit sections and complex cases, i.e., having several sources with different credentials
or even hosting several pipelines in the same project sharing the same config and credentials.

Expand Down Expand Up @@ -158,7 +158,7 @@ dlt.config["sheet_id"] = "23029402349032049"
dlt.secrets["destination.postgres.credentials"] = BaseHook.get_connection('postgres_dsn').extra
```

This will mock the `toml` provider to desired values.
This will mock the TOML provider to desired values.

## Example

Expand Down
2 changes: 1 addition & 1 deletion docs/website/docs/general-usage/credentials/setup.md
Original file line number Diff line number Diff line change
Expand Up @@ -237,7 +237,7 @@ The TOML provider also has the capability to read files from `~/.dlt/` (located
### Structure

`dlt` organizes sections in TOML files in a specific structure required by the [injection mechanism](advanced/#injection-mechanism).
Understanding this structure gives you more flexibility in setting credentials. For more details, see [Toml files structure](advanced/#toml-files-structure).
Understanding this structure gives you more flexibility in setting credentials. For more details, see [TOML files structure](advanced/#toml-files-structure).

## Custom providers

Expand Down
2 changes: 1 addition & 1 deletion docs/website/docs/running-in-production/running.md
Original file line number Diff line number Diff line change
Expand Up @@ -150,7 +150,7 @@ log_format="JSON"
- `json` to get the log in JSON format.
- [Python standard log format specifier](https://docs.python.org/3/library/logging.html#logrecord-attributes).

As with any other configuration, you can use environment variables instead of the `toml` file.
As with any other configuration, you can use environment variables instead of the TOML file.

- `RUNTIME__LOG_LEVEL` to set the log level.
- `LOG_FORMAT` to set the log format.
Expand Down
5 changes: 4 additions & 1 deletion docs/website/docs/walkthroughs/add_credentials.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,10 @@ project_id = "project_id" # please set me up!
private_key = "private_key" # please set me up!
client_email = "client_email" # please set me up!
```
> Note that for toml names are case-sensitive and sections are separated with ".".

:::note
Keys in TOML files are case-sensitive and sections are separated with a period (`.`).
:::

For destination credentials, read the [documentation pages for each destination](../dlt-ecosystem/destinations) to create and configure credentials.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@ If you are planning to run the pipeline with Google Cloud Composer, follow the n
pipedrive_api_key = "c66..."
```

> 💡 The `deploy` command will use an [Airflow variable](#4-add-credentials) called `dlt_secrets_toml` to store all the required secrets as a `toml` fragment. You can also use **environment variables** by passing the `--secrets-format env` option:
The `deploy` command will use an [Airflow variable](#4-add-credentials) called `dlt_secrets_toml` to store all the required secrets as a TOML fragment. You can also use **environment variables** by passing the `--secrets-format env` option:

```sh
dlt deploy pipedrive_pipeline.py airflow-composer --secrets-format env
Expand Down Expand Up @@ -384,15 +384,15 @@ There are two ways to pass the credentials:
will be displayed in the output:

```sh
3. Add the following toml-string to the Airflow UI as the dlt_secrets_toml variable.
3. Add the following TOML-string to the Airflow UI as the dlt_secrets_toml variable.

[sources.pipedrive]
pipedrive_api_key = "c66c..."
```

- Launch the Airflow UI, head to the **Admin** top-level menu, and select **Variables**.
- Add a new variable with the name `dlt_secrets_toml`.
- Paste the `toml` fragment displayed by the `deploy` command.
- Paste the TOML fragment displayed by the `deploy` command.
- 💡 The content of this variable will be used by the `dlt` Airflow helper instead of the local `secrets.toml` which you are familiar with. If your local secrets file contains anything else you want to access on Airflow, you are good to just copy the local `secrets.toml` content to the `dlt_secrets_toml` variable.

1. As environment variables.
Expand Down

0 comments on commit 543044e

Please sign in to comment.