From 4f4668902866bd2f1f431d6bf8b333e27679bcd6 Mon Sep 17 00:00:00 2001 From: Mark Raasveldt Date: Tue, 10 Sep 2024 12:30:10 +0200 Subject: [PATCH 1/4] Add secret information and new configuration flags for Postgres/MySQL --- docs/extensions/mysql.md | 92 +++++++++++++++++++++++++++++++++---- docs/extensions/postgres.md | 58 +++++++++++++++++++++++ 2 files changed, 141 insertions(+), 9 deletions(-) diff --git a/docs/extensions/mysql.md b/docs/extensions/mysql.md index 923f89b75aa..611930eb9c1 100644 --- a/docs/extensions/mysql.md +++ b/docs/extensions/mysql.md @@ -35,21 +35,95 @@ The connection string determines the parameters for how to connect to MySQL as a
-| Setting | Default | Environment variable | -|----------|----------------|----------------------| -| database | NULL | MYSQL_DATABASE | -| host | localhost | MYSQL_HOST | -| password | | MYSQL_PWD | -| port | 0 | MYSQL_TCP_PORT | -| socket | NULL | MYSQL_UNIX_PORT | -| user | ⟨current user⟩ | MYSQL_USER | +| Setting | Default | Environment variable | +|-------------|----------------|----------------------| +| database | NULL | MYSQL_DATABASE | +| host | localhost | MYSQL_HOST | +| password | | MYSQL_PWD | +| port | 0 | MYSQL_TCP_PORT | +| socket | NULL | MYSQL_UNIX_PORT | +| user | ⟨current user⟩ | MYSQL_USER | +| ssl_mode | preferred | | +| ssl_ca | | | +| ssl_capath | | | +| ssl_cert | | | +| ssl_cipher | | | +| ssl_crl | | | +| ssl_crlpath | | | +| ssl_key | | | + + +### Configuring via Secrets + +MySQL connection information can also be specified with [secrets](/docs/configuration/secrets_manager). The following syntax can be used to create a secret. + +```sql +CREATE SECRET ( + TYPE MYSQL, + HOST '127.0.0.1', + PORT 0, + DATABASE mysql, + USER 'mysql', + PASSWORD '' +); +``` + +The information from the secret will be used when `ATTACH` is called. We can leave the connection string empty to use all of the information stored in the secret. + +```sql +ATTACH '' AS mysql_db (TYPE MYSQL); +``` + +We can use the connection string to override individual options. For example, to connect to a different database while still using the same credentials, we can override only the database name in the following manner. + +```sql +ATTACH 'database=my_other_db' AS mysql_db (TYPE MYSQL); +``` + +By default, created secrets are temporary. Secrets can be persisted using the [`CREATE PERSISTENT SECRET command`](https://duckdb.org/docs/configuration/secrets_manager.html#persistent-secrets). Persistent secrets can be used across sessions. + +#### Managing Multiple Secrets + +Named secrets can be used to manage connections to multiple MySQL database instances. Secrets can be given a name upon creation. + +```sql +CREATE SECRET mysql_secret_one ( + TYPE MYSQL, + HOST '127.0.0.1', + PORT 0, + DATABASE mysql, + USER 'mysql', + PASSWORD '' +); +``` + +The secret can then be explicitly referenced using the `SECRET` parameter in the `ATTACH`. + +```sql +ATTACH '' AS mysql_db_one (TYPE MYSQL, SECRET mysql_secret_one); +``` + +### SSL Connections + +The [`ssl` connection parameters](https://dev.mysql.com/doc/refman/8.4/en/using-encrypted-connections.html) can be used to make SSL connections. Below is a description of the supported parameters. + +| Setting | Description | +|-------------|--------------------------------------------------------------------------------------------------------------------------------------------------| +| ssl_mode | The security state to use for the connection to the server: `disabled, required, verify_ca, verify_identity or preferred` (default: `preferred`) | +| ssl_ca | The path name of the Certificate Authority (CA) certificate file. | +| ssl_capath | The path name of the directory that contains trusted SSL CA certificate files. | +| ssl_cert | The path name of the client public key certificate file. | +| ssl_cipher | The list of permissible ciphers for SSL encryption. | +| ssl_crl | The path name of the file containing certificate revocation lists. | | +| ssl_crlpath | The path name of the directory that contains files containing certificate revocation lists. | | +| ssl_key | The path name of the client private key file. | | ### Reading MySQL Tables The tables in the MySQL database can be read as if they were normal DuckDB tables, but the underlying data is read directly from MySQL at query time. ```sql -SHOW TABLES; +SHOW ALL TABLES; ```
diff --git a/docs/extensions/postgres.md b/docs/extensions/postgres.md index 3d16ebc0c3d..76270770f49 100644 --- a/docs/extensions/postgres.md +++ b/docs/extensions/postgres.md @@ -35,6 +35,13 @@ To connect to the PostgreSQL instance with the given parameters in read-only mod ATTACH 'dbname=postgres user=postgres host=127.0.0.1' AS db (TYPE POSTGRES, READ_ONLY); ``` +By default, all schemas are attached. When working with large instances, it can be useful to only attach a specific schema. This can be accomplished using the `SCHEMA` command. + +```sql +ATTACH 'dbname=postgres user=postgres host=127.0.0.1' AS db (TYPE POSTGRES, SCHEMA 'public'); +``` + + ### Configuration The `ATTACH` command takes as input either a [`libpq` connection string](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING) @@ -59,6 +66,56 @@ host=localhost port=5432 dbname=mydb connect_timeout=10 An example URI is `postgresql://username@hostname/dbname`. +### Configuring via Secrets + +PostgreSQL connection information can also be specified with [secrets](/docs/configuration/secrets_manager). The following syntax can be used to create a secret. + +```sql +CREATE SECRET ( + TYPE POSTGRES, + HOST '127.0.0.1', + PORT 5432, + DATABASE postgres, + USER 'postgres', + PASSWORD '' +); +``` + +The information from the secret will be used when `ATTACH` is called. We can leave the Postgres connection string empty to use all of the information stored in the secret. + +```sql +ATTACH '' AS postgres_db (TYPE POSTGRES); +``` + +We can use the Postgres connection string to override individual options. For example, to connect to a different database while still using the same credentials, we can override only the database name in the following manner. + +```sql +ATTACH 'dbname=my_other_db' AS postgres_db (TYPE POSTGRES); +``` + +By default, created secrets are temporary. Secrets can be persisted using the [`CREATE PERSISTENT SECRET command`](https://duckdb.org/docs/configuration/secrets_manager.html#persistent-secrets). Persistent secrets can be used across sessions. + +#### Managing Multiple Secrets + +Named secrets can be used to manage connections to multiple Postgres database instances. Secrets can be given a name upon creation. + +```sql +CREATE SECRET postgres_secret_one ( + TYPE POSTGRES, + HOST '127.0.0.1', + PORT 5432, + DATABASE postgres, + USER 'postgres', + PASSWORD '' +); +``` + +The secret can then be explicitly referenced using the `SECRET` parameter in the `ATTACH`. + +```sql +ATTACH '' AS postgres_db_one (TYPE POSTGRES, SECRET postgres_secret_one); +``` + ### Configuring via Environment Variables PostgreSQL connection information can also be specified with [environment variables](https://www.postgresql.org/docs/current/libpq-envars.html). @@ -330,6 +387,7 @@ The extension exposes the following configuration parameters. | `pg_experimental_filter_pushdown` | Whether or not to use filter pushdown (currently experimental) | `false` | | `pg_pages_per_task` | The amount of pages per task | `1000` | | `pg_use_binary_copy` | Whether or not to use BINARY copy to read data | `true` | +| `pg_null_byte_replacement` | When writing NULL bytes to Postgres, replace them with the given character | `NULL` | | `pg_use_ctid_scan` | Whether or not to parallelize scanning using table ctids | `true` | ## Schema Cache From 7c4d3fe1d40e958c22296f23e7bd256cb3eabfe7 Mon Sep 17 00:00:00 2001 From: Mark Raasveldt Date: Tue, 10 Sep 2024 12:33:37 +0200 Subject: [PATCH 2/4] This snuck in --- docs/extensions/mysql.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/extensions/mysql.md b/docs/extensions/mysql.md index 611930eb9c1..c6ad7b22dc4 100644 --- a/docs/extensions/mysql.md +++ b/docs/extensions/mysql.md @@ -114,9 +114,9 @@ The [`ssl` connection parameters](https://dev.mysql.com/doc/refman/8.4/en/using- | ssl_capath | The path name of the directory that contains trusted SSL CA certificate files. | | ssl_cert | The path name of the client public key certificate file. | | ssl_cipher | The list of permissible ciphers for SSL encryption. | -| ssl_crl | The path name of the file containing certificate revocation lists. | | -| ssl_crlpath | The path name of the directory that contains files containing certificate revocation lists. | | -| ssl_key | The path name of the client private key file. | | +| ssl_crl | The path name of the file containing certificate revocation lists. | +| ssl_crlpath | The path name of the directory that contains files containing certificate revocation lists. | +| ssl_key | The path name of the client private key file. | ### Reading MySQL Tables From 5334d5609aef43709b0199f9630254b3660e35b4 Mon Sep 17 00:00:00 2001 From: Gabor Szarnyas Date: Tue, 10 Sep 2024 12:59:30 +0200 Subject: [PATCH 3/4] formatting --- docs/extensions/mysql.md | 2 +- docs/extensions/postgres.md | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/extensions/mysql.md b/docs/extensions/mysql.md index c6ad7b22dc4..1061c4d487e 100644 --- a/docs/extensions/mysql.md +++ b/docs/extensions/mysql.md @@ -80,7 +80,7 @@ We can use the connection string to override individual options. For example, to ATTACH 'database=my_other_db' AS mysql_db (TYPE MYSQL); ``` -By default, created secrets are temporary. Secrets can be persisted using the [`CREATE PERSISTENT SECRET command`](https://duckdb.org/docs/configuration/secrets_manager.html#persistent-secrets). Persistent secrets can be used across sessions. +By default, created secrets are temporary. Secrets can be persisted using the [`CREATE PERSISTENT SECRET` command]({% link docs/configuration/secrets_manager.md %}#persistent-secrets). Persistent secrets can be used across sessions. #### Managing Multiple Secrets diff --git a/docs/extensions/postgres.md b/docs/extensions/postgres.md index 76270770f49..70b27b318c2 100644 --- a/docs/extensions/postgres.md +++ b/docs/extensions/postgres.md @@ -47,7 +47,7 @@ ATTACH 'dbname=postgres user=postgres host=127.0.0.1' AS db (TYPE POSTGRES, SCHE The `ATTACH` command takes as input either a [`libpq` connection string](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING) or a [PostgreSQL URI](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING-URIS). -Below are some example connection strings and commonly used parameters. A full list of available parameters can be found [in the PostgreSQL documentation](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-PARAMKEYWORDS). +Below are some example connection strings and commonly used parameters. A full list of available parameters can be found in the [PostgreSQL documentation](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-PARAMKEYWORDS). ```text dbname=postgresscanner @@ -93,7 +93,7 @@ We can use the Postgres connection string to override individual options. For ex ATTACH 'dbname=my_other_db' AS postgres_db (TYPE POSTGRES); ``` -By default, created secrets are temporary. Secrets can be persisted using the [`CREATE PERSISTENT SECRET command`](https://duckdb.org/docs/configuration/secrets_manager.html#persistent-secrets). Persistent secrets can be used across sessions. +By default, created secrets are temporary. Secrets can be persisted using the [`CREATE PERSISTENT SECRET` command]({% link docs/configuration/secrets_manager.md %}#persistent-secrets). Persistent secrets can be used across sessions. #### Managing Multiple Secrets From ea7aaa5fe247a19054546c387879f4d232e1d9a9 Mon Sep 17 00:00:00 2001 From: Gabor Szarnyas Date: Tue, 10 Sep 2024 13:01:23 +0200 Subject: [PATCH 4/4] Bump Rouge version --- Gemfile.lock | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Gemfile.lock b/Gemfile.lock index ce879b30e69..1a6f40c5259 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -10,7 +10,7 @@ GIT GIT remote: https://github.com/duckdb/rouge.git - revision: d487f892d0caa08e0ea6053509aa83b765c60cd1 + revision: 0ca2c48ea4da4419f7089988c8663a9bf8c75064 branch: duckdb specs: rouge (3.3823.1)