From 2598d44a6e8dbb18712317c9b5210b5eed7fc5cd Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Wed, 20 Mar 2024 15:14:13 +0800 Subject: [PATCH] dm: support customized secret key (#16740) --- TOC.md | 1 + dm/dm-command-line-flags.md | 18 +++++---------- dm/dm-customized-secret-key.md | 36 ++++++++++++++++++++++++++++++ dm/dm-export-import-config.md | 2 +- dm/dm-manage-source.md | 6 ++++- dm/dm-master-configuration-file.md | 5 ++++- dm/dmctl-introduction.md | 2 -- dm/maintain-dm-using-tiup.md | 6 ++++- dm/quick-start-create-source.md | 2 ++ dm/quick-start-create-task.md | 4 ++++ dm/quick-start-with-dm.md | 2 +- 11 files changed, 65 insertions(+), 19 deletions(-) create mode 100644 dm/dm-customized-secret-key.md diff --git a/TOC.md b/TOC.md index f76d89486e3c6..f1e45a1f520dd 100644 --- a/TOC.md +++ b/TOC.md @@ -446,6 +446,7 @@ - [Binlog Event Filter](/dm/dm-binlog-event-filter.md) - [Filter DMLs Using SQL Expressions](/dm/feature-expression-filter.md) - [Online DDL Tool Support](/dm/dm-online-ddl-tool-support.md) + - [Customize a Secret Key for Encryption and Decryption](/dm/dm-customized-secret-key.md) - Manage a Data Migration Task - [Precheck a Task](/dm/dm-precheck.md) - [Create a Task](/dm/dm-create-task.md) diff --git a/dm/dm-command-line-flags.md b/dm/dm-command-line-flags.md index e9712359c3c35..33759c9305483 100644 --- a/dm/dm-command-line-flags.md +++ b/dm/dm-command-line-flags.md @@ -75,6 +75,12 @@ This document introduces DM's command-line flags. - The default value is `"http://127.0.0.1:8291"` - Required flag +### `--secret-key-path` + +- The path of the customized secret key for encryption and decryption +- The default value is `""` +- Optional flag + ## DM-worker ### `--advertise-addr` @@ -132,15 +138,3 @@ This document introduces DM's command-line flags. - The `{advertise-addr}` of any DM-master node in the cluster to be connected by dmctl - The default value is `""` - It is a required flag when dmctl interacts with DM-master - -### `--encrypt` - -- Encrypts the plaintext database password into ciphertext -- The default value is `""` -- When this flag is specified, it is only used to encrypt the plaintext without interacting with the DM-master - -### `--decrypt` - -- Decrypts ciphertext encrypted with dmctl into plaintext -- The default value is `""` -- When this flag is specified, it is only used to decrypt the ciphertext without interacting with the DM-master diff --git a/dm/dm-customized-secret-key.md b/dm/dm-customized-secret-key.md new file mode 100644 index 0000000000000..af71bb0bdf4bc --- /dev/null +++ b/dm/dm-customized-secret-key.md @@ -0,0 +1,36 @@ +--- +title: Customize a Secret Key for DM Encryption and Decryption +summary: Learn how to customize a secret key to encrypt and decrypt passwords used in the DM（Data Migration）data source and migration task configurations. +--- + +# Customize a Secret Key for DM Encryption and Decryption + +Before v8.0.0, [DM](/dm/dm-overview.md) uses a [fixed AES-256 secret key](https://github.com/pingcap/tiflow/blob/1252979421fc83ffa2a1548d981e505f7fc0b909/dm/pkg/encrypt/encrypt.go#L27) to encrypt and decrypt passwords in the data source and migration task configurations. However, using a fixed secret key might pose security risks, especially in environments where security is crucial. To enhance security, starting from v8.0.0, DM removes the fixed secret key and enables you to customize a secret key. + +## Usage + +1. Create a custom key file, which must contain a 64-character hexadecimal AES-256 secret key. +2. In the DM-master [command-line flags](/dm/dm-command-line-flags.md) or [configuration file](/dm/dm-master-configuration-file.md), specify `secret-key-path` as the path of your custom key file. + +## Upgrade from a version earlier than v8.0.0 + +Because DM no longer uses the fixed secret key starting from v8.0.0, pay attention to the following when upgrading DM from versions earlier than v8.0.0: + +- If plaintext passwords are used in both [data source configurations](/dm/dm-source-configuration-file.md) and [migration task configurations](/dm/task-configuration-file-full.md), no additional steps are required for the upgrade. +- If encrypted passwords are used in [data source configurations](/dm/dm-source-configuration-file.md) and [migration task configurations](/dm/task-configuration-file-full.md) or if you want to use encrypted passwords in the future, you need to do the following: + 1. Add the `secret-key-path` parameter to the [DM-master configuration file](/dm/dm-master-configuration-file.md) and specify it as the path of your custom key file. The file must contain a 64-character hexadecimal AES-256 key. If the [fixed AES-256 secret key](https://github.com/pingcap/tiflow/blob/1252979421fc83ffa2a1548d981e505f7fc0b909/dm/pkg/encrypt/encrypt.go#L27) was used for encryption before upgrading, you can copy this secret key to your key file. Make sure all DM-master nodes use the same secret key configuration. + 2. Perform a rolling upgrade of DM-master first, followed by a rolling upgrade of DM-worker. For more information, see [Rolling upgrade](/dm/maintain-dm-using-tiup.md#rolling-upgrade). + +## Update the secret key for encryption and decryption + +To update the secret key used for encryption and decryption, take the following steps: + +1. Update `secret-key-path` in the [DM-master configuration file](/dm/dm-master-configuration-file.md). + + > **Note:** + > + > - Make sure all DM-master nodes are updated to the same secret key configuration. + > - During the secret key update, do not create new [data source configuration files](/dm/dm-source-configuration-file.md) or [migration task configuration files](/dm/task-configuration-file-full.md). + +2. Perform a rolling restart of DM-master. +3. Use the passwords encrypted with `tiup dmctl encrypt` (dmctl version >= v8.0.0) when you create new [data source configuration files](/dm/dm-source-configuration-file.md) and [migration task configuration files](/dm/task-configuration-file-full.md). \ No newline at end of file diff --git a/dm/dm-export-import-config.md b/dm/dm-export-import-config.md index a67d4095a8345..50d6f609f8b80 100644 --- a/dm/dm-export-import-config.md +++ b/dm/dm-export-import-config.md @@ -9,7 +9,7 @@ summary: Learn how to export and import data sources and task configuration of c > **Note:** > -> For clusters earlier than v2.0.5, you can use dmctl v2.0.5 or later to export and import the data source and task configuration files. +> For clusters earlier than v2.0.5, you can use dmctl (>= v2.0.5 and < v8.0.0) to export and import the data source and task configuration files. {{< copyable "" >}} diff --git a/dm/dm-manage-source.md b/dm/dm-manage-source.md index 511df3caa1986..c84223c314c61 100644 --- a/dm/dm-manage-source.md +++ b/dm/dm-manage-source.md @@ -11,10 +11,14 @@ This document introduces how to manage data source configurations, including enc In DM configuration files, it is recommended to use the password encrypted with dmctl. For one original password, the encrypted password is different after each encryption. +> **Note:** +> +> Starting from v8.0.0, you must configure [`secret-key-path`](/dm/dm-master-configuration-file.md) for DM-master before using the `dmctl encrypt` command. + {{< copyable "shell-regular" >}} ```bash -./dmctl -encrypt 'abc!@#123' +./dmctl encrypt 'abc!@#123' ``` ``` diff --git a/dm/dm-master-configuration-file.md b/dm/dm-master-configuration-file.md index d32617c5b591f..ec58677c0ab18 100644 --- a/dm/dm-master-configuration-file.md +++ b/dm/dm-master-configuration-file.md @@ -34,7 +34,9 @@ join = "" ssl-ca = "/path/to/ca.pem" ssl-cert = "/path/to/cert.pem" ssl-key = "/path/to/key.pem" -cert-allowed-cn = ["dm"] +cert-allowed-cn = ["dm"] + +secret-key-path = "/path/to/secret/key" ``` ## Configuration parameters @@ -58,3 +60,4 @@ This section introduces the configuration parameters of DM-master. | `ssl-cert` | The path of the file that contains X509 certificate in PEM format for DM-master to connect with other components. | | `ssl-key` | The path of the file that contains X509 key in PEM format for DM-master to connect with other components. | | `cert-allowed-cn` | Common Name list. | +| `secret-key-path` | The file path of the secret key, which is used to encrypt and decrypt upstream and downstream passwords. The file must contain a 64-character hexadecimal AES-256 secret key. | \ No newline at end of file diff --git a/dm/dmctl-introduction.md b/dm/dmctl-introduction.md index a14f4598b1d76..b3d479de9957d 100644 --- a/dm/dmctl-introduction.md +++ b/dm/dmctl-introduction.md @@ -45,7 +45,6 @@ Available Commands: binlog-schema manage or show table schema in schema tracker check-task Checks the configuration file of the task config manage config operations - decrypt Decrypts cipher text to plain text encrypt Encrypts plain text to cipher text help Gets help about any command list-member Lists member information @@ -98,7 +97,6 @@ Available Commands: binlog-schema manage or show table schema in schema tracker check-task Checks the configuration file of the task config manage config operations - decrypt Decrypts cipher text to plain text encrypt Encrypts plain text to cipher text help Gets help about any command list-member Lists member information diff --git a/dm/maintain-dm-using-tiup.md b/dm/maintain-dm-using-tiup.md index 49d50316ae334..5707edc917d46 100644 --- a/dm/maintain-dm-using-tiup.md +++ b/dm/maintain-dm-using-tiup.md @@ -183,7 +183,7 @@ For example, to scale out a DM-worker node in the `prod-cluster` cluster, take t > > Before upgrading, you can use `config export` to export the configuration files of clusters. After upgrading, if you need to downgrade to an earlier version, you can first redeploy the earlier cluster and then use `config import` to import the previous configuration files. > -> For clusters earlier than v2.0.5, you can use dmctl v2.0.5 or later to export and import the data source and task configuration files. +> For clusters earlier than v2.0.5, you can use dmctl (>= v2.0.5 and < v8.0.0) to export and import the data source and task configuration files. > > For clusters later than v2.0.2, currently, it is not supported to automatically import the configuration related to relay worker. You can use `start-relay` command to manually [start relay log](/dm/relay-log.md#enable-and-disable-relay-log). @@ -193,6 +193,10 @@ The rolling upgrade process is made as transparent as possible to the applicatio You can run the `tiup dm upgrade` command to upgrade a DM cluster. For example, the following command upgrades the cluster to `${version}`. Modify `${version}` to your needed version before running this command: +> **Note:** +> +> Starting from v8.0.0, DM removes the fixed secret key for encryption and decryption and enables you to customize a secret key for encryption and decryption. If encrypted passwords are used in [data source configurations](/dm/dm-source-configuration-file.md) and [migration task configurations](/dm/task-configuration-file-full.md) before the upgrade, you need to refer to the upgrade steps in [Customize a Secret Key for DM Encryption and Decryption](/dm/dm-customized-secret-key.md) for additional operations. + {{< copyable "shell-regular" >}} ```bash diff --git a/dm/quick-start-create-source.md b/dm/quick-start-create-source.md index 23886f4eaf8b9..a3af12981f8a6 100644 --- a/dm/quick-start-create-source.md +++ b/dm/quick-start-create-source.md @@ -19,6 +19,8 @@ A data source contains the information for accessing the upstream migration task In DM configuration files, it is recommended to use the password encrypted with dmctl. You can follow the example below to obtain the encrypted password of the data source, which can be used to write the configuration file later. + Starting from v8.0.0, you must configure [`secret-key-path`](/dm/dm-master-configuration-file.md) for DM-master before using the `tiup dmctl encrypt` command. + {{< copyable "shell-regular" >}} ```bash diff --git a/dm/quick-start-create-task.md b/dm/quick-start-create-task.md index 42828fbf32a9b..e4888d3dab432 100644 --- a/dm/quick-start-create-task.md +++ b/dm/quick-start-create-task.md @@ -97,6 +97,10 @@ Before starting a data migration task, you need to configure the MySQL data sour For safety reasons, it is recommended to configure and use encrypted passwords. You can use dmctl to encrypt the MySQL/TiDB password. Suppose the password is "123456": +> **Note:** +> +> Starting from v8.0.0, you must configure [`secret-key-path`](/dm/dm-master-configuration-file.md) for DM-master before using the `dmctl encrypt` command. + {{< copyable "shell-regular" >}} ```bash diff --git a/dm/quick-start-with-dm.md b/dm/quick-start-with-dm.md index 75231181e152d..26c30a4a2b9d9 100644 --- a/dm/quick-start-with-dm.md +++ b/dm/quick-start-with-dm.md @@ -48,7 +48,7 @@ You can use one or multiple MySQL instances as an upstream data source. from: host: "127.0.0.1" user: "root" - password: "fCxfQ9XKCezSzuCD0Wf5dUD+LsKegSg=" # encrypt with `tiup dmctl --encrypt "123456"` + password: "fCxfQ9XKCezSzuCD0Wf5dUD+LsKegSg=" port: 3306 ```