dm: support customized secret key (#16740)

Author: qiancai

TOC.md

@@ -446,6 +446,7 @@
       - [Binlog Event Filter](/dm/dm-binlog-event-filter.md)
       - [Filter DMLs Using SQL Expressions](/dm/feature-expression-filter.md)
       - [Online DDL Tool Support](/dm/dm-online-ddl-tool-support.md)
+      - [Customize a Secret Key for Encryption and Decryption](/dm/dm-customized-secret-key.md)
       - Manage a Data Migration Task
         - [Precheck a Task](/dm/dm-precheck.md)
         - [Create a Task](/dm/dm-create-task.md)

dm/dm-command-line-flags.md

@@ -75,6 +75,12 @@ This document introduces DM's command-line flags.
 - The default value is `"http://127.0.0.1:8291"`
 - Required flag
 
+### `--secret-key-path`
+
+- The path of the customized secret key for encryption and decryption
+- The default value is `""`
+- Optional flag
+
 ## DM-worker
 
 ### `--advertise-addr`
@@ -132,15 +138,3 @@ This document introduces DM's command-line flags.
 - The `{advertise-addr}` of any DM-master node in the cluster to be connected by dmctl
 - The default value is `""`
 - It is a required flag when dmctl interacts with DM-master
-
-### `--encrypt`
-
-- Encrypts the plaintext database password into ciphertext
-- The default value is `""`
-- When this flag is specified, it is only used to encrypt the plaintext without interacting with the DM-master
-
-### `--decrypt`
-
-- Decrypts ciphertext encrypted with dmctl into plaintext
-- The default value is `""`
-- When this flag is specified, it is only used to decrypt the ciphertext without interacting with the DM-master

dm/dm-customized-secret-key.md

@@ -0,0 +1,36 @@
+---
+title: Customize a Secret Key for DM Encryption and Decryption
+summary: Learn how to customize a secret key to encrypt and decrypt passwords used in the DM（Data Migration）data source and migration task configurations.
+---
+
+# Customize a Secret Key for DM Encryption and Decryption
+
+Before v8.0.0, [DM](/dm/dm-overview.md) uses a [fixed AES-256 secret key](https://github.com/pingcap/tiflow/blob/1252979421fc83ffa2a1548d981e505f7fc0b909/dm/pkg/encrypt/encrypt.go#L27) to encrypt and decrypt passwords in the data source and migration task configurations. However, using a fixed secret key might pose security risks, especially in environments where security is crucial. To enhance security, starting from v8.0.0, DM removes the fixed secret key and enables you to customize a secret key.
+
+## Usage
+
+1. Create a custom key file, which must contain a 64-character hexadecimal AES-256 secret key.
+2. In the DM-master [command-line flags](/dm/dm-command-line-flags.md) or [configuration file](/dm/dm-master-configuration-file.md), specify `secret-key-path` as the path of your custom key file.
+
+## Upgrade from a version earlier than v8.0.0
+
+Because DM no longer uses the fixed secret key starting from v8.0.0, pay attention to the following when upgrading DM from versions earlier than v8.0.0:
+
+- If plaintext passwords are used in both [data source configurations](/dm/dm-source-configuration-file.md) and [migration task configurations](/dm/task-configuration-file-full.md), no additional steps are required for the upgrade.
+- If encrypted passwords are used in [data source configurations](/dm/dm-source-configuration-file.md) and [migration task configurations](/dm/task-configuration-file-full.md) or if you want to use encrypted passwords in the future, you need to do the following:
+    1. Add the `secret-key-path` parameter to the [DM-master configuration file](/dm/dm-master-configuration-file.md) and specify it as the path of your custom key file. The file must contain a 64-character hexadecimal AES-256 key. If the [fixed AES-256 secret key](https://github.com/pingcap/tiflow/blob/1252979421fc83ffa2a1548d981e505f7fc0b909/dm/pkg/encrypt/encrypt.go#L27) was used for encryption before upgrading, you can copy this secret key to your key file. Make sure all DM-master nodes use the same secret key configuration.
+    2. Perform a rolling upgrade of DM-master first, followed by a rolling upgrade of DM-worker. For more information, see [Rolling upgrade](/dm/maintain-dm-using-tiup.md#rolling-upgrade).
+
+## Update the secret key for encryption and decryption
+
+To update the secret key used for encryption and decryption, take the following steps:
+
+1. Update `secret-key-path` in the [DM-master configuration file](/dm/dm-master-configuration-file.md).
+
+    > **Note:**
+    >
+    > - Make sure all DM-master nodes are updated to the same secret key configuration.
+    > - During the secret key update, do not create new [data source configuration files](/dm/dm-source-configuration-file.md) or [migration task configuration files](/dm/task-configuration-file-full.md).
+
+2. Perform a rolling restart of DM-master.
+3. Use the passwords encrypted with `tiup dmctl encrypt` (dmctl version >= v8.0.0) when you create new [data source configuration files](/dm/dm-source-configuration-file.md) and [migration task configuration files](/dm/task-configuration-file-full.md).

dm/dm-export-import-config.md

@@ -9,7 +9,7 @@ summary: Learn how to export and import data sources and task configuration of c
 
 > **Note:**
 >
-> For clusters earlier than v2.0.5, you can use dmctl v2.0.5 or later to export and import the data source and task configuration files.
+> For clusters earlier than v2.0.5, you can use dmctl (>= v2.0.5 and < v8.0.0) to export and import the data source and task configuration files.
 
 {{< copyable "" >}}
 

dm/dm-manage-source.md

@@ -11,10 +11,14 @@ This document introduces how to manage data source configurations, including enc
 
 In DM configuration files, it is recommended to use the password encrypted with dmctl. For one original password, the encrypted password is different after each encryption.
 
+> **Note:**
+>
+> Starting from v8.0.0, you must configure [`secret-key-path`](/dm/dm-master-configuration-file.md) for DM-master before using the `dmctl encrypt` command.
+
 {{< copyable "shell-regular" >}}
 
 ```bash
-./dmctl -encrypt 'abc!@#123'
+./dmctl encrypt 'abc!@#123'
 ```
 
 ```

dm/dm-master-configuration-file.md

@@ -34,7 +34,9 @@ join = ""
 ssl-ca = "/path/to/ca.pem"
 ssl-cert = "/path/to/cert.pem"
 ssl-key = "/path/to/key.pem"
-cert-allowed-cn = ["dm"] 
+cert-allowed-cn = ["dm"]
+
+secret-key-path = "/path/to/secret/key"
 ```
 
 ## Configuration parameters
@@ -58,3 +60,4 @@ This section introduces the configuration parameters of DM-master.
 | `ssl-cert` | The path of the file that contains X509 certificate in PEM format for DM-master to connect with other components. |
 | `ssl-key` | The path of the file that contains X509 key in PEM format for DM-master to connect with other components. |
 | `cert-allowed-cn` | Common Name list. |
+| `secret-key-path` | The file path of the secret key, which is used to encrypt and decrypt upstream and downstream passwords. The file must contain a 64-character hexadecimal AES-256 secret key. |

dm/dmctl-introduction.md

@@ -45,7 +45,6 @@ Available Commands:
   binlog-schema   manage or show table schema in schema tracker
   check-task      Checks the configuration file of the task
   config          manage config operations
-  decrypt         Decrypts cipher text to plain text
   encrypt         Encrypts plain text to cipher text
   help            Gets help about any command
   list-member     Lists member information
@@ -98,7 +97,6 @@ Available Commands:
   binlog-schema   manage or show table schema in schema tracker
   check-task      Checks the configuration file of the task
   config          manage config operations
-  decrypt         Decrypts cipher text to plain text
   encrypt         Encrypts plain text to cipher text
   help            Gets help about any command
   list-member     Lists member information

dm/maintain-dm-using-tiup.md

@@ -183,7 +183,7 @@ For example, to scale out a DM-worker node in the `prod-cluster` cluster, take t
 >
 > Before upgrading, you can use `config export` to export the configuration files of clusters. After upgrading, if you need to downgrade to an earlier version, you can first redeploy the earlier cluster and then use `config import` to import the previous configuration files.
 >
-> For clusters earlier than v2.0.5, you can use dmctl v2.0.5 or later to export and import the data source and task configuration files.
+> For clusters earlier than v2.0.5, you can use dmctl (>= v2.0.5 and < v8.0.0) to export and import the data source and task configuration files.
 >
 > For clusters later than v2.0.2, currently, it is not supported to automatically import the configuration related to relay worker. You can use `start-relay` command to manually [start relay log](/dm/relay-log.md#enable-and-disable-relay-log).
 
@@ -193,6 +193,10 @@ The rolling upgrade process is made as transparent as possible to the applicatio
 
 You can run the `tiup dm upgrade` command to upgrade a DM cluster. For example, the following command upgrades the cluster to `${version}`. Modify `${version}` to your needed version before running this command:
 
+> **Note:**
+>
+> Starting from v8.0.0, DM removes the fixed secret key for encryption and decryption and enables you to customize a secret key for encryption and decryption. If encrypted passwords are used in [data source configurations](/dm/dm-source-configuration-file.md) and [migration task configurations](/dm/task-configuration-file-full.md) before the upgrade, you need to refer to the upgrade steps in [Customize a Secret Key for DM Encryption and Decryption](/dm/dm-customized-secret-key.md) for additional operations.
+
 {{< copyable "shell-regular" >}}
 
 ```bash

dm/quick-start-create-source.md

@@ -19,6 +19,8 @@ A data source contains the information for accessing the upstream migration task
 
     In DM configuration files, it is recommended to use the password encrypted with dmctl. You can follow the example below to obtain the encrypted password of the data source, which can be used to write the configuration file later.
 
+    Starting from v8.0.0, you must configure [`secret-key-path`](/dm/dm-master-configuration-file.md) for DM-master before using the `tiup dmctl encrypt` command.
+
     {{< copyable "shell-regular" >}}
 
     ```bash

dm/quick-start-create-task.md

@@ -97,6 +97,10 @@ Before starting a data migration task, you need to configure the MySQL data sour
 
 For safety reasons, it is recommended to configure and use encrypted passwords. You can use dmctl to encrypt the MySQL/TiDB password. Suppose the password is "123456":
 
+> **Note:**
+>
+> Starting from v8.0.0, you must configure [`secret-key-path`](/dm/dm-master-configuration-file.md) for DM-master before using the `dmctl encrypt` command.
+
 {{< copyable "shell-regular" >}}
 
 ```bash

dm/quick-start-with-dm.md

@@ -48,7 +48,7 @@ You can use one or multiple MySQL instances as an upstream data source.
     from:
       host: "127.0.0.1"
       user: "root"
-      password: "fCxfQ9XKCezSzuCD0Wf5dUD+LsKegSg="  # encrypt with `tiup dmctl --encrypt "123456"`
+      password: "fCxfQ9XKCezSzuCD0Wf5dUD+LsKegSg="
       port: 3306
     ```