Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

br: add doc for log backup encryption #18906

Open
wants to merge 10 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
123 changes: 109 additions & 14 deletions br/br-pitr-manual.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ Each subcommand is described as follows:
- `tiup br log truncate`: clean up the log backup data from the backup storage.
- `tiup br log metadata`: query the metadata of the log backup data.

### Start a backup task
### Start a log backup task

You can run the `tiup br log start` command to start a log backup task. This task runs in the background of your TiDB cluster and automatically backs up the change log of KV storage to the backup storage.

Expand Down Expand Up @@ -83,13 +83,76 @@ The example output only shows the common parameters. These parameters are descri
Usage example:

```shell
tiup br log start --task-name=pitr --pd="${PD_IP}:2379" \
--storage='s3://backup-101/logbackup?access-key=${access-key}&secret-access-key=${secret-access-key}"'
tiup br log start \
--task-name=pitr \
--pd="${PD_IP}:2379" \
--storage='s3://backup-101/logbackup?access-key=${access-key}&secret-access-key=${secret-access-key}'
```
qiancai marked this conversation as resolved.
Show resolved Hide resolved

### Encrypt the log backup data

> **Warning:**
>
> Currently, this feature is experimental. It is not recommended that you use it in production environments. If you find a bug, you can report an [issue](https://github.com/pingcap/tidb/issues) on GitHub.

BR enables you to encrypt log backup data before uploading it to your backup storage.

Starting from TiDB v8.4.0, you can encrypt log backup data by passing the following parameters in the log backup command, which is similar to [snapshot backup encryption](/br/br-snapshot-manual.md#encrypt-the-backup-data):

- `--log.crypter.method`: Encryption algorithm, which can be `aes128-ctr`, `aes192-ctr`, or `aes256-ctr`. The default value is `plaintext`, indicating that data is not encrypted.
- `--log.crypter.key`: Encryption key in hexadecimal string format. It is a 128-bit (16 bytes) key for the algorithm `aes128-ctr`, a 24-byte key for the algorithm `aes192-ctr`, and a 32-byte key for the algorithm `aes256-ctr`.
- `--log.crypter.key-file`: The key file. You can directly pass in the file path where the key is stored as a parameter without passing in the `crypter.key`.

Tristan1900 marked this conversation as resolved.
Show resolved Hide resolved
The following is an example:

```shell
tiup br log start \
--task-name=pitr-with-encryption
--pd ${PD_IP}:2379 \
--storage "s3://${BACKUP_COLLECTION_ADDR}/snapshot-${DATE}?access-key=${AWS_ACCESS_KEY}&secret-access-key=${AWS_SECRET_ACCESS_KEY}" \
--log.crypter.method aes128-ctr \
--log.crypter.key 0123456789abcdef0123456789abcdef
```

However, in scenarios with higher security requirements, you might not want to pass a fixed encryption key directly in the command line. To further enhance security, you can use a master key based encryption system to manage encryption keys. This system generates different data keys to encrypt different log backup files and supports master key rotation. You can configure it using the following parameters:

- `--master-key-crypter-method`: Encryption algorithm based on the master key, which can be `aes128-ctr`, `aes192-ctr`, or `aes256-ctr`. The default value is `plaintext`, indicating that data is not encrypted.
- `--master-key`: Master key configuration. It can be a master key stored on a local disk or a master key managed by a cloud Key Management Service (KMS).

Encrypt using a master key stored on a local disk:

Tristan1900 marked this conversation as resolved.
Show resolved Hide resolved
```shell
tiup br log start \
--task-name=pitr-with-encryption \
--pd ${PD_IP}:2379 \
--storage "s3://${BACKUP_COLLECTION_ADDR}/snapshot-${DATE}?access-key=${AWS_ACCESS_KEY}&secret-access-key=${AWS_SECRET_ACCESS_KEY}" \
--master-key-crypter-method aes128-ctr \
--master-key "local:///path/to/master.key"
```

Encrypt using a master key managed by AWS KMS:

```shell
...
--master-key "aws-kms:///${AWS_KMS_KEY_ID}?AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY}&AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}&REGION=${AWS_REGION}"
Comment on lines +136 to +137
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
...
--master-key "aws-kms:///${AWS_KMS_KEY_ID}?AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY}&AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}&REGION=${AWS_REGION}"
tiup br log start \
--task-name=pitr-with-encryption \
--pd ${PD_IP}:2379 \
--storage "s3://${BACKUP_COLLECTION_ADDR}/snapshot-${DATE}?access-key=${AWS_ACCESS_KEY}&secret-access-key=${AWS_SECRET_ACCESS_KEY}" \
--master-key-crypter-method aes128-ctr \
--master-key "aws-kms:///${AWS_KMS_KEY_ID}?AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY}&AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}&REGION=${AWS_REGION}"

```

qiancai marked this conversation as resolved.
Show resolved Hide resolved
### Query the backup status
Encrypt using a master key managed by Google Cloud KMS:

You can run the `tiup br log status` command to query the backup status.
```shell
...
--master-key "gcp-kms:///projects/$GCP_PROJECT_ID/locations/$GCP_LOCATION/keyRings/$GCP_KEY_RING/cryptoKeys/$GCP_KEY_NAME?AUTH=specified&CREDENTIALS=$GCP_CREDENTIALS_PATH"

Comment on lines +143 to +145
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
...
--master-key "gcp-kms:///projects/$GCP_PROJECT_ID/locations/$GCP_LOCATION/keyRings/$GCP_KEY_RING/cryptoKeys/$GCP_KEY_NAME?AUTH=specified&CREDENTIALS=$GCP_CREDENTIALS_PATH"
tiup br log start \
--task-name=pitr-with-encryption \
--pd ${PD_IP}:2379 \
--storage "s3://${BACKUP_COLLECTION_ADDR}/snapshot-${DATE}?access-key=${AWS_ACCESS_KEY}&secret-access-key=${AWS_SECRET_ACCESS_KEY}" \
--master-key-crypter-method aes128-ctr \
--master-key "gcp-kms:///projects/$GCP_PROJECT_ID/locations/$GCP_LOCATION/keyRings/$GCP_KEY_RING/cryptoKeys/$GCP_KEY_NAME?AUTH=specified&CREDENTIALS=$GCP_CREDENTIALS_PATH"

```

> **Note:**
>
> - If the key is lost, the log backup data cannot be restored to the cluster.
> - The encryption feature needs to be used on `br` and TiDB clusters v8.4.0 or later versions. The encrypted log backup data cannot be restored on clusters earlier than v8.4.0.

### Query the log backup status

You can run the `tiup br log status` command to query the log backup status.

Run `tiup br log status --help` to see the help information:

Expand Down Expand Up @@ -144,9 +207,9 @@ The output fields are described as follows:
- `checkpoint [global]`: all data before this checkpoint is backed up to the backup storage. This is the latest timestamp available for restoring the backup data.
- `error [store]`: the error the log backup program encounters on the storage node.

### Pause and resume a backup task
### Pause and resume a log backup task

You can run the `tiup br log pause` command to pause a running backup task.
You can run the `tiup br log pause` command to pause a running log backup task.

Run `tiup br log pause --help` to see the help information:

Expand Down Expand Up @@ -210,11 +273,11 @@ Usage example:
tiup br log resume --task-name=pitr --pd="${PD_IP}:2379"
```

### Stop and restart a backup task
### Stop and restart a log backup task

You can stop a log backup task by running the `tiup br log stop` command and restart a backup task that is stopped by using the original `--storage` directory.
You can stop a log backup task by running the `tiup br log stop` command and restart a log backup task that is stopped by using the original `--storage` directory.

#### Stop a backup task
### Stop a log backup task

You can run the `tiup br log stop` command to stop a log backup task. This command cleans up the task metadata in the backup cluster.

Expand Down Expand Up @@ -248,15 +311,15 @@ Usage example:
tiup br log stop --task-name=pitr --pd="${PD_IP}:2379"
```

#### Restart a backup task
#### Restart a log backup task

After running the `tiup br log stop` command to stop a log backup task, you can create a new log backup task in another `--storage` directory or restart the log backup task in the original `--storage` directory by running the `tiup br log start` command. If you restart the task in the original `--storage` directory, pay attention to the following points:

- Parameters of the `--storage` directory for restarting a task must be the same as the task that is stopped.
- The `--start-ts` does not need to be specified. BR automatically starts the backup from the last backup checkpoint.
- If the task is stopped for a long time and multiple versions of the data have been garbage collected, the error `BR:Backup:ErrBackupGCSafepointExceeded` is reported when you attempt to restart the task. In this case, you have to create a new log backup task in another `--storage` directory.

### Clean up backup data
### Clean up log backup data

You can run the `tiup br log truncate` command to clean up the outdated or no longer needed log backup data.

Expand Down Expand Up @@ -303,9 +366,9 @@ Clearing data files... DONE; take = 43.504161ms, kv-count = 53, kv-size = 4573(4
Removing metadata... DONE; take = 24.038962ms
```

### View the backup metadata
### View the log backup metadata

You can run the `tiup br log metadata` command to view the backup metadata in the storage system, such as the earliest and latest timestamp that can be restored.
You can run the `tiup br log metadata` command to view the log backup metadata in the storage system, such as the earliest and latest timestamp that can be restored.

Run `tiup br log metadata --help` to see the help information:

Expand Down Expand Up @@ -399,3 +462,35 @@ Restore KV Files <--------------------------------------------------------------
> - When you restore the cluster for the first time, you must specify the full snapshot data. Otherwise, some data in the newly created table might be incorrect due to rewriting Table ID rules. For more information, see GitHub issue [#54418](https://github.com/pingcap/tidb/issues/54418).
> - You cannot restore the log backup data of a certain time period repeatedly. If you restore the log backup data of a range `[t1=10, t2=20)` repeatedly, the restored data might be inconsistent.
> - When you restore log data of different time periods in multiple batches, ensure that the log data is restored in consecutive order. If you restore the log backup data of `[t1, t2)`, `[t2, t3)`, and `[t3, t4)` in consecutive order, the restored data is consistent. However, if you restore `[t1, t2)` and then skip `[t2, t3)` to restore `[t3, t4)`, the restored data might be inconsistent.

### Restore encrypted log backup data

> **Warning:**
>
> Currently, this feature is experimental. It is not recommended that you use it in production environments. If you find a bug, you can report an [issue](https://github.com/pingcap/tidb/issues) on GitHub.

To restore encrypted log backup data, you need to pass the corresponding decryption parameters in the restore command. Make sure that the decryption parameters are the same as those used for encryption. If the decryption algorithm or key is incorrect, the data cannot be restored.

The following is an example:

```shell
tiup br restore point --pd="${PD_IP}:2379"
--storage='s3://backup-101/logbackup?access-key=${ACCESS-KEY}&secret-access-key=${SECRET-ACCESS-KEY}"'
--full-backup-storage='s3://backup-101/snapshot-202205120000?access-key=${ACCESS-KEY}&secret-access-key=${SECRET-ACCESS-KEY}"'
--crypter.method aes128-ctr
--crypter.key 0123456789abcdef0123456789abcdef
--log.crypter.method aes128-ctr
--log.crypter.key 0123456789abcdef0123456789abcdef
```

If a log backup is encrypted using a master key, you can decrypt and restore the backup data using the following command:

```shell
tiup br restore point --pd="${PD_IP}:2379"
--storage='s3://backup-101/logbackup?access-key=${ACCESS-KEY}&secret-access-key=${SECRET-ACCESS-KEY}"'
--full-backup-storage='s3://backup-101/snapshot-202205120000?access-key=${ACCESS-KEY}&secret-access-key=${SECRET-ACCESS-KEY}"'
--crypter.method aes128-ctr
--crypter.key 0123456789abcdef0123456789abcdef
--master-key-crypter-method aes128-ctr
--master-key "local:///path/to/master.key"
```
12 changes: 6 additions & 6 deletions br/use-br-command-line-tool.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,12 +77,12 @@ To back up cluster data, run the `tiup br backup` command. You can add the `full

To start log backup and manage log backup tasks, run the `tiup br log` command.

- [Start a log backup task](/br/br-pitr-manual.md#start-a-backup-task)
- [Query the backup status](/br/br-pitr-manual.md#query-the-backup-status)
- [Pause and resume a log backup task](/br/br-pitr-manual.md#pause-and-resume-a-backup-task)
- [Stop and restart a log backup task](/br/br-pitr-manual.md#stop-and-restart-a-backup-task)
- [Clean up the backup data](/br/br-pitr-manual.md#clean-up-backup-data)
- [View the backup metadata](/br/br-pitr-manual.md#view-the-backup-metadata)
- [Start a log backup task](/br/br-pitr-manual.md#start-a-log-backup-task)
- [Query the log backup status](/br/br-pitr-manual.md#query-the-log-backup-status)
- [Pause and resume a log backup task](/br/br-pitr-manual.md#pause-and-resume-a-log-backup-task)
- [Stop and restart a log backup task](/br/br-pitr-manual.md#stop-and-restart-a-log-backup-task)
- [Clean up the backup data](/br/br-pitr-manual.md#clean-up-log-backup-data)
- [View the backup metadata](/br/br-pitr-manual.md#view-the-log-backup-metadata)

## Commands of restoring backup data

Expand Down
Loading