Merge pull request #3601 from Mytherin/postgresdocs

Add secret information and new configuration flags for Postgres/MySQL
duckdb · Sep 10, 2024 · 65c92c4 · 65c92c4
2 parents a612caa + ea7aaa5
commit 65c92c4
Show file tree

Hide file tree

Showing 3 changed files with 143 additions and 11 deletions.
diff --git a/Gemfile.lock b/Gemfile.lock
@@ -10,7 +10,7 @@ GIT
 
 GIT
   remote: https://github.com/duckdb/rouge.git
-  revision: d487f892d0caa08e0ea6053509aa83b765c60cd1
+  revision: 0ca2c48ea4da4419f7089988c8663a9bf8c75064
   branch: duckdb
   specs:
     rouge (3.3823.1)

diff --git a/docs/extensions/mysql.md b/docs/extensions/mysql.md
@@ -35,21 +35,95 @@ The connection string determines the parameters for how to connect to MySQL as a
 
 <div class="narrow_table monospace_table"></div>
 
-| Setting  |    Default     | Environment variable |
-|----------|----------------|----------------------|
-| database | NULL           | MYSQL_DATABASE       |
-| host     | localhost      | MYSQL_HOST           |
-| password |                | MYSQL_PWD            |
-| port     | 0              | MYSQL_TCP_PORT       |
-| socket   | NULL           | MYSQL_UNIX_PORT      |
-| user     | ⟨current user⟩ | MYSQL_USER           |
+| Setting     | Default        | Environment variable |
+|-------------|----------------|----------------------|
+| database    | NULL           | MYSQL_DATABASE       |
+| host        | localhost      | MYSQL_HOST           |
+| password    |                | MYSQL_PWD            |
+| port        | 0              | MYSQL_TCP_PORT       |
+| socket      | NULL           | MYSQL_UNIX_PORT      |
+| user        | ⟨current user⟩ | MYSQL_USER           |
+| ssl_mode    | preferred      |                      |
+| ssl_ca      |                |                      |
+| ssl_capath  |                |                      |
+| ssl_cert    |                |                      |
+| ssl_cipher  |                |                      |
+| ssl_crl     |                |                      |
+| ssl_crlpath |                |                      |
+| ssl_key     |                |                      |
+
+
+### Configuring via Secrets
+
+MySQL connection information can also be specified with [secrets](/docs/configuration/secrets_manager). The following syntax can be used to create a secret.
+
+```sql
+CREATE SECRET (
+    TYPE MYSQL,
+    HOST '127.0.0.1',
+    PORT 0,
+    DATABASE mysql,
+    USER 'mysql',
+    PASSWORD ''
+);
+```
+
+The information from the secret will be used when `ATTACH` is called. We can leave the connection string empty to use all of the information stored in the secret.
+
+```sql
+ATTACH '' AS mysql_db (TYPE MYSQL);
+```
+
+We can use the connection string to override individual options. For example, to connect to a different database while still using the same credentials, we can override only the database name in the following manner.
+
+```sql
+ATTACH 'database=my_other_db' AS mysql_db (TYPE MYSQL);
+```
+
+By default, created secrets are temporary. Secrets can be persisted using the [`CREATE PERSISTENT SECRET` command]({% link docs/configuration/secrets_manager.md %}#persistent-secrets). Persistent secrets can be used across sessions.
+
+#### Managing Multiple Secrets
+
+Named secrets can be used to manage connections to multiple MySQL database instances. Secrets can be given a name upon creation.
+
+```sql
+CREATE SECRET mysql_secret_one (
+    TYPE MYSQL,
+    HOST '127.0.0.1',
+    PORT 0,
+    DATABASE mysql,
+    USER 'mysql',
+    PASSWORD ''
+);
+```
+
+The secret can then be explicitly referenced using the `SECRET` parameter in the `ATTACH`.
+
+```sql
+ATTACH '' AS mysql_db_one (TYPE MYSQL, SECRET mysql_secret_one);
+```
+
+### SSL Connections
+
+The [`ssl` connection parameters](https://dev.mysql.com/doc/refman/8.4/en/using-encrypted-connections.html) can be used to make SSL connections. Below is a description of the supported parameters.
+
+| Setting     | Description                                                                                                                                      |
+|-------------|--------------------------------------------------------------------------------------------------------------------------------------------------|
+| ssl_mode    | The security state to use for the connection to the server: `disabled, required, verify_ca, verify_identity or preferred` (default: `preferred`) |
+| ssl_ca      | The path name of the Certificate Authority (CA) certificate file.                                                                                |
+| ssl_capath  | The path name of the directory that contains trusted SSL CA certificate files.                                                                   |
+| ssl_cert    | The path name of the client public key certificate file.                                                                                         |
+| ssl_cipher  | The list of permissible ciphers for SSL encryption.                                                                                              |
+| ssl_crl     | The path name of the file containing certificate revocation lists.                                                                               |
+| ssl_crlpath | The path name of the directory that contains files containing certificate revocation lists.                                                      |
+| ssl_key     | The path name of the client private key file.                                                                                                    |
 
 ### Reading MySQL Tables
 
 The tables in the MySQL database can be read as if they were normal DuckDB tables, but the underlying data is read directly from MySQL at query time.
 
 ```sql
-SHOW TABLES;
+SHOW ALL TABLES;
 ```
 
 <div class="narrow_table monospace_table"></div>

diff --git a/docs/extensions/postgres.md b/docs/extensions/postgres.md
@@ -35,12 +35,19 @@ To connect to the PostgreSQL instance with the given parameters in read-only mod
 ATTACH 'dbname=postgres user=postgres host=127.0.0.1' AS db (TYPE POSTGRES, READ_ONLY);
 ```
 
+By default, all schemas are attached. When working with large instances, it can be useful to only attach a specific schema. This can be accomplished using the `SCHEMA` command.
+
+```sql
+ATTACH 'dbname=postgres user=postgres host=127.0.0.1' AS db (TYPE POSTGRES, SCHEMA 'public');
+```
+
+
 ### Configuration
 
 The `ATTACH` command takes as input either a [`libpq` connection string](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING)
 or a [PostgreSQL URI](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING-URIS).
 
-Below are some example connection strings and commonly used parameters. A full list of available parameters can be found [in the PostgreSQL documentation](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-PARAMKEYWORDS).
+Below are some example connection strings and commonly used parameters. A full list of available parameters can be found in the [PostgreSQL documentation](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-PARAMKEYWORDS).
 
 ```text
 dbname=postgresscanner
@@ -59,6 +66,56 @@ host=localhost port=5432 dbname=mydb connect_timeout=10
 
 An example URI is `postgresql://username@hostname/dbname`.
 
+### Configuring via Secrets
+
+PostgreSQL connection information can also be specified with [secrets](/docs/configuration/secrets_manager). The following syntax can be used to create a secret.
+
+```sql
+CREATE SECRET (
+	TYPE POSTGRES,
+	HOST '127.0.0.1',
+    PORT 5432,
+	DATABASE postgres,
+    USER 'postgres',
+	PASSWORD ''
+);
+```
+
+The information from the secret will be used when `ATTACH` is called. We can leave the Postgres connection string empty to use all of the information stored in the secret.
+
+```sql
+ATTACH '' AS postgres_db (TYPE POSTGRES);
+```
+
+We can use the Postgres connection string to override individual options. For example, to connect to a different database while still using the same credentials, we can override only the database name in the following manner.
+
+```sql
+ATTACH 'dbname=my_other_db' AS postgres_db (TYPE POSTGRES);
+```
+
+By default, created secrets are temporary. Secrets can be persisted using the [`CREATE PERSISTENT SECRET` command]({% link docs/configuration/secrets_manager.md %}#persistent-secrets). Persistent secrets can be used across sessions.
+
+#### Managing Multiple Secrets
+
+Named secrets can be used to manage connections to multiple Postgres database instances. Secrets can be given a name upon creation.
+
+```sql
+CREATE SECRET postgres_secret_one (
+	TYPE POSTGRES,
+	HOST '127.0.0.1',
+    PORT 5432,
+	DATABASE postgres,
+    USER 'postgres',
+	PASSWORD ''
+);
+```
+
+The secret can then be explicitly referenced using the `SECRET` parameter in the `ATTACH`.
+
+```sql
+ATTACH '' AS postgres_db_one (TYPE POSTGRES, SECRET postgres_secret_one);
+```
+
 ### Configuring via Environment Variables
 
 PostgreSQL connection information can also be specified with [environment variables](https://www.postgresql.org/docs/current/libpq-envars.html).
@@ -330,6 +387,7 @@ The extension exposes the following configuration parameters.
 | `pg_experimental_filter_pushdown` | Whether or not to use filter pushdown (currently experimental)               | `false` |
 | `pg_pages_per_task`               | The amount of pages per task                                                 | `1000`  |
 | `pg_use_binary_copy`              | Whether or not to use BINARY copy to read data                               | `true`  |
+| `pg_null_byte_replacement`        | When writing NULL bytes to Postgres, replace them with the given character   | `NULL`  |
 | `pg_use_ctid_scan`                | Whether or not to parallelize scanning using table ctids                     | `true`  |
 
 ## Schema Cache