Skip to content

Commit

Permalink
encryption: add tikv kms encryption for gcp and azure (#16761)
Browse files Browse the repository at this point in the history
  • Loading branch information
@qiancai
qiancai authored Mar 20, 2024
1 parent 2598d44 commit 1e14467
Showing 1 changed file with 118 additions and 16 deletions.
134 changes: 118 additions & 16 deletions encryption-at-rest.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ When a TiDB cluster is deployed, the majority of user data is stored on TiKV and

TiKV supports encryption at rest. This feature allows TiKV to transparently encrypt data files using [AES](https://en.wikipedia.org/wiki/Advanced_Encryption_Standard) or [SM4](https://en.wikipedia.org/wiki/SM4_(cipher)) in [CTR](https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation) mode. To enable encryption at rest, an encryption key must be provided by the user and this key is called master key. TiKV automatically rotates data keys that it used to encrypt actual data files. Manually rotating the master key can be done occasionally. Note that encryption at rest only encrypts data at rest (namely, on disk) and not while data is transferred over network. It is advised to use TLS together with encryption at rest.

Optionally, you can use AWS KMS for both cloud and self-hosted deployments. You can also supply the plaintext master key in a file.
You can use Key Management Service (KMS) for both cloud and self-hosted deployments or supply the plaintext master key in a file.

TiKV currently does not exclude encryption keys and user data from core dumps. It is advised to disable core dumps for the TiKV process when using encryption at rest. This is not currently handled by TiKV itself.

Expand Down Expand Up @@ -59,17 +59,48 @@ TiKV currently supports encrypting data using AES128, AES192, AES256, or SM4 (on
* Master key. The master key is provided by user and is used to encrypt the data keys TiKV generates. Management of master key is external to TiKV.
* Data key. The data key is generated by TiKV and is the key actually used to encrypt data.

The same master key can be shared by multiple instances of TiKV. The recommended way to provide a master key in production is via AWS KMS. Create a customer master key (CMK) through AWS KMS, and then provide the CMK key ID to TiKV in the configuration file. The TiKV process needs access to the KMS CMK while it is running, which can be done by using an [IAM role](https://aws.amazon.com/iam/). If TiKV fails to get access to the KMS CMK, it will fail to start or restart. Refer to AWS documentation for [KMS](https://docs.aws.amazon.com/kms/index.html) and [IAM](https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html) usage.
The same master key can be shared by multiple instances of TiKV. The recommended way to provide a master key in production is via KMS. Currently, TiKV supports KMS encryption on [AWS](https://docs.aws.amazon.com/kms/index.html), [Google Cloud](https://cloud.google.com/security/products/security-key-management?hl=en), and [Azure](https://learn.microsoft.com/en-us/azure/key-vault/). To enable KMS encryption, you need to create a customer master key (CMK) through KMS, and then provide the CMK key ID to TiKV using the configuration file. If TiKV fails to get access to the KMS CMK, it will fail to start or restart.

Alternatively, if using custom key is desired, supplying the master key via file is also supported. The file must contain a 256 bits (or 32 bytes) key encoded as hex string, end with a newline (namely, `\n`), and contain nothing else. Persisting the key on disk, however, leaks the key, so the key file is only suitable to be stored on the `tempfs` in RAM.

Data keys are passed to the underlying storage engine (namely, RocksDB). All files written by RocksDB, including SST files, WAL files, and the MANIFEST file, are encrypted by the current data key. Other temporary files used by TiKV that may include user data are also encrypted using the same data key. Data keys are automatically rotated by TiKV every week by default, but the period is configurable. On key rotation, TiKV does not rewrite all existing files to replace the key, but RocksDB compaction are expected to rewrite old data into new data files, with the most recent data key, if the cluster gets constant write workload. TiKV keeps track of the key and encryption method used to encrypt each of the files and use the information to decrypt the content on reads.

Regardless of data encryption method, data keys are encrypted using AES256 in GCM mode for additional authentication. This required the master key to be 256 bits (32 bytes), when passing from file instead of KMS.

### Key creation
### Configure encryption

To enable encryption, you can add the encryption section in the configuration files of TiKV and PD:

```
[security.encryption]
data-encryption-method = "aes128-ctr"
data-key-rotation-period = "168h" # 7 days
```

- `data-encryption-method` specifies the encryption algorithm. The possible values are `"aes128-ctr"`, `"aes192-ctr"`, `"aes256-ctr"`, `"sm4-ctr"` (only for v6.3.0 and later versions), and `"plaintext"`. The default value is `"plaintext"`, which means that encryption is disabled by default.

- For a new TiKV cluster or an existing TiKV cluster, only data written after encryption has been enabled is guaranteed to be encrypted.
- To disable encryption after it is enabled, remove `data-encryption-method` from the configuration file or set its value to `"plaintext"`, and then restart TiKV.
- To change the encryption algorithm, replace the value of `data-encryption-method` with a supported encryption algorithm, and then restart TiKV. After the replacement, as new data is written in, the encryption files generated by the previous encryption algorithm are gradually rewritten to files generated by the new encryption algorithm.

- `data-key-rotation-period` specifies how often TiKV rotates keys.

If encryption is enabled (that is, the value of `data-encryption-method` is not `"plaintext"`), you must specify a master key in either of the following ways:

- [Specify a master key via KMS](#specify-a-master-key-via-kms)
- [Specify a master key via a file](#specify-a-master-key-via-a-file)

To create a key on AWS, follow these steps:
#### Specify a master key via KMS

TiKV supports KMS encryption for three platforms: AWS, Google Cloud, and Azure. Depending on the platform where your service is deployed, you can choose one of them to configure KMS encryption.

<SimpleTab>

<div label="AWS KMS">

**Step 1. Create a master key**

To create a key on AWS, take the following steps:

1. Go to the [AWS KMS](https://console.aws.amazon.com/kms) on the AWS console.
2. Make sure that you have selected the correct region on the top right corner of your console.
Expand All @@ -85,19 +116,9 @@ aws --region us-west-2 kms create-alias --alias-name "alias/tidb-tde" --target-k

The `--target-key-id` to enter in the second command is in the output of the first command.

### Configure encryption
**Step 2. Configure the master key**

To enable encryption, you can add the encryption section in the configuration files of TiKV and PD:

```
[security.encryption]
data-encryption-method = "aes128-ctr"
data-key-rotation-period = "168h" # 7 days
```

Possible values for `data-encryption-method` are "aes128-ctr", "aes192-ctr", "aes256-ctr", "sm4-ctr" (only in v6.3.0 and later versions) and "plaintext". The default value is "plaintext", which means encryption is not turned on. `data-key-rotation-period` defines how often TiKV rotates the data key. Encryption can be turned on for a fresh TiKV cluster, or an existing TiKV cluster, though only data written after encryption is enabled is guaranteed to be encrypted. To disable encryption, remove `data-encryption-method` in the configuration file, or reset it to "plaintext", and restart TiKV. To change encryption method, update `data-encryption-method` in the configuration file and restart TiKV. To change the encryption algorithm, replace `data-encryption-method` with a supported encryption algorithm and then restart TiKV. After the replacement, as new data is written in, the encryption file generated by the previous encryption algorithm is gradually rewritten to a file generated by the new encryption algorithm.

The master key has to be specified if encryption is enabled (that is,`data-encryption-method` is not "plaintext"). To specify a AWS KMS CMK as master key, add the `encryption.master-key` section after the `encryption` section:
To specify the master key using AWS KMS, add the `[security.encryption.master-key]` configuration after the `[security.encryption]` section in the TiKV configuration file:

```
[security.encryption.master-key]
Expand All @@ -111,6 +132,87 @@ The `key-id` specifies the key ID for the KMS CMK. The `region` is the AWS regio

You can also use [multi-Region keys](https://docs.aws.amazon.com/kms/latest/developerguide/multi-region-keys-overview.html) in AWS. For this, you need to set up a primary key in a specific region and add replica keys in the regions you require.

</div>
<div label="Google Cloud KMS">

**Step 1. Create a master key**

To create a key on Google Cloud, take the following steps:

1. Go to the [Key Management](https://console.cloud.google.com/security/kms/keyrings) page in the Google Cloud console.
2. Click **Create key ring**. Enter a name for the key ring, select a location of the key ring, and then click **Create**. Note that the location of the key ring needs to cover the region where the TiDB cluster is deployed.
3. Select the key ring you created in the previous step, and then click **Create Key** on the key ring details page.
4. Enter a name for the key, set the key information as follows, and then click **Create**.

- **Protection level**: **Software** or **HSM**
- **Key Material**: **Generated key**
- **Purpose**: **Symmetric encrypt/decrypt**

You can also perform this operation using the gcloud CLI:

```shell
gcloud kms keyrings create "key-ring-name" --location "global"
gcloud kms keys create "key-name" --keyring "key-ring-name" --location "global" --purpose "encryption" --rotation-period "30d"
```

Make sure to replace the values of `"key-ring-name"`, `"key-name"`, `"global"`, and `"30d"` in the preceding command with the names and configurations corresponding to your actual key.

**Step 2. Configure the master key**

To specify the master key using Google Cloud KMS, add the `[security.encryption.master-key]` configuration after the `[security.encryption]` section:

```
[security.encryption.master-key]
type = "kms"
key-id = "projects/project-name/locations/global/keyRings/key-ring-name/cryptoKeys/key-name"
vendor = "gcp"
[security.encryption.master-key.gcp]
credential-file-path = "/path/to/credential.json"
```

- `key-id` specifies the key ID of the KMS CMK.
- `credential-file-path` specifies the path of the authentication credentials file, which currently supports two types of credentials: Service Account and Authentication User. If the TiKV environment is already configured with [application default credentials](https://cloud.google.com/docs/authentication/application-default-credentials), there is no need to configure `credential-file-path`.

</div>
<div label="Azure KMS">

**Step 1. Create a master key**

To create a key on Azure, refer to the instructions in [Set and retrieve a key from Azure Key Vault using the Azure portal](https://learn.microsoft.com/en-us/azure/key-vault/keys/quick-create-portal).

**Step 2. Configure the master key**

To specify the master key using Azure KMS, add the `[security.encryption.master-key]` configuration after the `[security.encryption]` section in the TiKV configuration file:

```
[security.encryption.master-key]
type = 'kms'
key-id = 'your-kms-key-id'
region = 'region-name'
endpoint = 'endpoint'
vendor = 'azure'
[security.encryption.master-key.azure]
tenant-id = 'tenant_id'
client-id = 'client_id'
keyvault-url = 'keyvault_url'
hsm-name = 'hsm_name'
hsm-url = 'hsm_url'
# The following four fields are optional, used to set client authentication credentials. You can configure them according to the requirements of your scenario.
client_certificate = ""
client_certificate_path = ""
client_certificate_password = ""
client_secret = ""
```

Except `vendor`, you need to modify the values of other fields in the preceding configuration to the corresponding configuration of the actual key.

</div>
</SimpleTab>

#### Specify a master key via a file

To specify a master key that's stored in a file, the master key configuration would look like the following:

```
Expand Down

0 comments on commit 1e14467

Please sign in to comment.