Skip to content

Commit

Permalink
Update serverless-external-storage.md
Browse files Browse the repository at this point in the history
  • Loading branch information
@hfxsd
hfxsd committed Aug 27, 2024
1 parent d18be3f commit 06341eb
Showing 1 changed file with 97 additions and 95 deletions.
192 changes: 97 additions & 95 deletions tidb-cloud/serverless-external-storage.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,13 +5,13 @@ summary: Learn how to configure Amazon Simple Storage Service (Amazon S3) access

# Configure External Storage Access for TiDB Serverless

If you want import data from or export data to external storage in TiDB Serverless, you need to configure cross-account access. This document describes how to configure access to an external storage, including Amazon Simple Storage Service (Amazon S3), Google Cloud Storage (GCS) and Azure Blob Storage for TiDB Serverless.
If you want import data from or export data to an external storage in TiDB Serverless, you need to configure cross-account access. This document describes how to configure access to an external storage, including Amazon Simple Storage Service (Amazon S3), Google Cloud Storage (GCS) and Azure Blob Storage for TiDB Serverless.

If you need to configure these external storages for TiDB Dedicated, see [Configure Amazon S3 Access and GCS Access for TiDB Dedicated](/tidb-cloud/config-s3-and-gcs-access.md).

## Configure Amazon S3 access

To allow TiDB Serverless to access your Amazon S3 bucket, you need to configure the bucket access for TiDB Serverless. You can use either of the following methods to configure the bucket access:
To allow a iDB Serverless cluster to access your Amazon S3 bucket, you need to configure the bucket access for TiDB Serverless. You can use either of the following methods to configure the bucket access:

- Use an AWS access key: use the access key of an IAM user to access your Amazon S3 bucket.
- Use a Role ARN: use a Role ARN to access your Amazon S3 bucket.
Expand All @@ -25,31 +25,31 @@ Take the following steps to create one:

1. Open the **Import & Export** page for your target cluster.

1. Log in to the [TiDB Cloud console](https://tidbcloud.com/) and navigate to the [**Clusters**](https://tidbcloud.com/console/clusters) page of your project.
1. Log in to the [TiDB Cloud console](https://tidbcloud.com/) and navigate to the [**Clusters**](https://tidbcloud.com/console/clusters) page of your project.

2. Click the name of your target cluster to go to its overview page, and then click **Import & Export** in the left navigation pane.
2. Click the name of your target cluster to go to its overview page, and then click **Import & Export** in the left navigation pane.

2. Open the `Add New ARN` pop-up window.

1. Click **Import from S3** or **Export to Amazon S3** according to your needs.
1. Click **Import from S3** or **Export to Amazon S3** according to your needs.

2. Fill in the `Folder URI` or `File URI` according to the instructions.
2. Fill in the `Folder URI` or `File URI` according to the instructions.

3. Choose **AWS Role ARN** and click **Click here to create new one with AWS CloudFormation** to open the **Add New ARN** pop-up window.
3. Choose **AWS Role ARN** and click **Click here to create new one with AWS CloudFormation** to open the **Add New ARN** pop-up window.

3. Create a role ARN with AWS CloudFormation Template.

1. Click **AWS Console with CloudFormation Template** in the **Add New ARN** pop-up window.
1. Click **AWS Console with CloudFormation Template** in the **Add New ARN** pop-up window.

2. Log in to the AWS and you will be redirected to the AWS CloudFormation Template page.
2. Log in to the AWS and you will be redirected to the AWS CloudFormation Template page.

3. Fill in the **Role Name**.
3. Fill in the **Role Name**.

4. Acknowledge to create a new role and click **Create stack** to create the role ARN.
4. Acknowledge to create a new role and click **Create stack** to create the role ARN.

5. After the CloudFormation stack is executed, you can find the Role ARN value in the **Outputs** tab.
5. After the CloudFormation stack is executed, you can find the Role ARN value in the **Outputs** tab.

![img.png](/media/tidb-cloud/serverless-external-storage/serverless-role-arn.png)
![img.png](/media/tidb-cloud/serverless-external-storage/serverless-role-arn.png)


</div>
Expand All @@ -64,95 +64,95 @@ Take the following steps to create one:

2. In the AWS Management Console, create a managed policy for your Amazon S3 bucket.

1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/).
1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/).

2. In the **Buckets** list, choose the name of your bucket with the source data, and then click **Copy ARN** to get your S3 bucket ARN (for example, `arn:aws:s3:::tidb-cloud-source-data`). Take a note of the bucket ARN for later use.
2. In the **Buckets** list, choose the name of your bucket with the source data, and then click **Copy ARN** to get your S3 bucket ARN (for example, `arn:aws:s3:::tidb-cloud-source-data`). Take a note of the bucket ARN for later use.

![Copy bucket ARN](/media/tidb-cloud/copy-bucket-arn.png)
![Copy bucket ARN](/media/tidb-cloud/copy-bucket-arn.png)

3. Open the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/), click **Policies** in the navigation pane on the left, and then click **Create Policy**.
3. Open the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/), click **Policies** in the navigation pane on the left, and then click **Create Policy**.

![Create a policy](/media/tidb-cloud/aws-create-policy.png)
![Create a policy](/media/tidb-cloud/aws-create-policy.png)

4. On the **Create policy** page, click the **JSON** tab.
4. On the **Create policy** page, click the **JSON** tab.

5. Copy the following access policy template and paste it to the policy text field.

```json
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:GetObjectVersion",
"s3:PutObject"
],
"Resource": "<Your S3 bucket ARN>/<Directory of your source data>/*"
},
{
"Sid": "VisualEditor1",
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:GetBucketLocation"
],
"Resource": "<Your S3 bucket ARN>"
}
]
}
```

In the policy text field, update the following configurations to your own values.

- `"Resource": "<Your S3 bucket ARN>/<Directory of the source data>/*"`

For example, if your source data is stored in the root directory of the `tidb-cloud-source-data` bucket, use `"Resource": "arn:aws:s3:::tidb-cloud-source-data/*"`. If your source data is stored in the `mydata` directory of the bucket, use `"Resource": "arn:aws:s3:::tidb-cloud-source-data/mydata/*"`. Make sure that `/*` is added to the end of the directory so TiDB Cloud can access all files in this directory.

- `"Resource": "<Your S3 bucket ARN>"`

For example, `"Resource": "arn:aws:s3:::tidb-cloud-source-data"`.

- If you have enabled AWS Key Management Service key (SSE-KMS) with customer-managed key encryption, make sure the following configuration is included in the policy. `"arn:aws:kms:ap-northeast-1:105880447796:key/c3046e91-fdfc-4f3a-acff-00597dd3801f"` is a sample KMS key of the bucket.

```
{
"Sid": "AllowKMSkey",
"Effect": "Allow",
"Action": [
"kms:Decrypt"
],
"Resource": "arn:aws:kms:ap-northeast-1:105880447796:key/c3046e91-fdfc-4f3a-acff-00597dd3801f"
}
```

If the objects in your bucket have been copied from another encrypted bucket, the KMS key value needs to include the keys of both buckets. For example, `"Resource": ["arn:aws:kms:ap-northeast-1:105880447796:key/c3046e91-fdfc-4f3a-acff-00597dd3801f","arn:aws:kms:ap-northeast-1:495580073302:key/0d7926a7-6ecc-4bf7-a9c1-a38f0faec0cd"]`.

6. Click **Next: Tags**, add a tag of the policy (optional), and then click **Next:Review**.

7. Set a policy name, and then click **Create policy**.
5. Copy the following access policy template and paste it to the policy text field.

```json
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:GetObjectVersion",
"s3:PutObject"
],
"Resource": "<Your S3 bucket ARN>/<Directory of your source data>/*"
},
{
"Sid": "VisualEditor1",
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:GetBucketLocation"
],
"Resource": "<Your S3 bucket ARN>"
}
]
}
```

In the policy text field, update the following configurations to your own values.

- `"Resource": "<Your S3 bucket ARN>/<Directory of the source data>/*"`

For example, if your source data is stored in the root directory of the `tidb-cloud-source-data` bucket, use `"Resource": "arn:aws:s3:::tidb-cloud-source-data/*"`. If your source data is stored in the `mydata` directory of the bucket, use `"Resource": "arn:aws:s3:::tidb-cloud-source-data/mydata/*"`. Make sure that `/*` is added to the end of the directory so TiDB Cloud can access all files in this directory.

- `"Resource": "<Your S3 bucket ARN>"`

For example, `"Resource": "arn:aws:s3:::tidb-cloud-source-data"`.

- If you have enabled AWS Key Management Service key (SSE-KMS) with customer-managed key encryption, make sure the following configuration is included in the policy. `"arn:aws:kms:ap-northeast-1:105880447796:key/c3046e91-fdfc-4f3a-acff-00597dd3801f"` is a sample KMS key of the bucket.

```
{
"Sid": "AllowKMSkey",
"Effect": "Allow",
"Action": [
"kms:Decrypt"
],
"Resource": "arn:aws:kms:ap-northeast-1:105880447796:key/c3046e91-fdfc-4f3a-acff-00597dd3801f"
}
```

- If the objects in your bucket have been copied from another encrypted bucket, the KMS key value needs to include the keys of both buckets. For example, `"Resource": ["arn:aws:kms:ap-northeast-1:105880447796:key/c3046e91-fdfc-4f3a-acff-00597dd3801f","arn:aws:kms:ap-northeast-1:495580073302:key/0d7926a7-6ecc-4bf7-a9c1-a38f0faec0cd"]`.

6. Click **Next: Tags**, add a tag of the policy (optional), and then click **Next:Review**.

7. Set a policy name, and then click **Create policy**.

3. In the AWS Management Console, create an access role for TiDB Cloud and get the role ARN.

1. In the [IAM console](https://console.aws.amazon.com/iam/), click **Roles** in the navigation pane on the left, and then click **Create role**.
1. In the [IAM console](https://console.aws.amazon.com/iam/), click **Roles** in the navigation pane on the left, and then click **Create role**.

![Create a role](/media/tidb-cloud/aws-create-role.png)
![Create a role](/media/tidb-cloud/aws-create-role.png)

2. To create a role, fill in the following information:
2. To create a role, fill in the following information:

- Under **Trusted entity type**, select **AWS account**.
- Under **An AWS account**, select **Another AWS account**, and then paste the TiDB Cloud account ID to the **Account ID** field.
- Under **Options**, click **Require external ID (Best practice when a third party will assume this role)**, and then paste the TiDB Cloud External ID to the **External ID** field. If the role is created without "Require external ID", once the configuration is done for one TiDB cluster in a project, all TiDB clusters in that project can use the same Role ARN to access your Amazon S3 bucket. If the role is created with the account ID and external ID, only the corresponding TiDB cluster can access the bucket.
- Under **Trusted entity type**, select **AWS account**.
- Under **An AWS account**, select **Another AWS account**, and then paste the TiDB Cloud account ID to the **Account ID** field.
- Under **Options**, click **Require external ID (Best practice when a third party will assume this role)**, and then paste the TiDB Cloud External ID to the **External ID** field. If the role is created without "Require external ID", once the configuration is done for one TiDB cluster in a project, all TiDB clusters in that project can use the same Role ARN to access your Amazon S3 bucket. If the role is created with the account ID and external ID, only the corresponding TiDB cluster can access the bucket.

3. Click **Next** to open the policy list, choose the policy you just created, and then click **Next**.
3. Click **Next** to open the policy list, choose the policy you just created, and then click **Next**.

4. Under **Role details**, set a name for the role, and then click **Create role** in the lower-right corner. After the role is created, the list of roles is displayed.
4. Under **Role details**, set a name for the role, and then click **Create role** in the lower-right corner. After the role is created, the list of roles is displayed.

5. In the list of roles, click the name of the role that you just created to go to its summary page, and then you can get the role ARN.
5. In the list of roles, click the name of the role that you just created to go to its summary page, and then you can get the role ARN.

![Copy AWS role ARN](/media/tidb-cloud/aws-role-arn.png)
![Copy AWS role ARN](/media/tidb-cloud/aws-role-arn.png)

</div>

Expand Down Expand Up @@ -182,15 +182,16 @@ To allow TiDB Serverless to access your GCS bucket, you need to configure the GC
Take the following steps to configure a service account key:

1. Create a service account with needed permission in the Google Cloud [service account page](https://console.cloud.google.com/iam-admin/serviceaccounts). For more information, see [Creating a service account](https://cloud.google.com/iam/docs/creating-managing-service-accounts).
1. Enter a service account name.
2. Enter a description of the service account (Optional).
3. Click **CREATE AND CONTINUE** to create the service account and continue the next step.
4. In the `Grant this service account access to project`, choose the [IAM roles](https://cloud.google.com/iam/docs/understanding-roles) with needed permission. For example, TiDB Serverless export needs a role with `storage.objects.create` permission.
5. Click **Continue** to the next step.
6. Optional: In the `Grant users access to this service account`, choose members that need to [attach the service account to other resources](https://cloud.google.com/iam/docs/attach-service-accounts).
7. Click **Done** to finish creating the service account.

![img.png](/media/tidb-cloud/serverless-external-storage/gcs-service-account.png)
1. Enter a service account name.
2. Enter a description of the service account (Optional).
3. Click **CREATE AND CONTINUE** to create the service account and continue the next step.
4. In the `Grant this service account access to project`, choose the [IAM roles](https://cloud.google.com/iam/docs/understanding-roles) with needed permission. For example, TiDB Serverless export needs a role with `storage.objects.create` permission.
5. Click **Continue** to the next step.
6. Optional: In the `Grant users access to this service account`, choose members that need to [attach the service account to other resources](https://cloud.google.com/iam/docs/attach-service-accounts).
7. Click **Done** to finish creating the service account.

![img.png](/media/tidb-cloud/serverless-external-storage/gcs-service-account.png)

2. Click the service account and then click the **ADD KEY** button in the `KEYS` page to create a service account key.

Expand All @@ -217,12 +218,13 @@ Take the following steps to configure a service SAS token:
![img.png](/media/tidb-cloud/serverless-external-storage/azure-sas.png)

3. In the `Shared access signature` page, create a service SAS token with needed permissions as follows. For more information, see [Create a service SAS token](https://docs.microsoft.com/en-us/azure/storage/common/storage-sas-overview).

1. Choose the `Blob` service under the `Allowed services` section.
2. Choose the `Container` and `Object` under the `Allowed Resource types` section.
3. Choose the permission under the `Allowed permissions` section as you needed. For example: TiDB Serverless exports needs `Read` and `Write` permission.
4. Adjust the `Start and expiry date/time` as you needed.
5. You can keep other settings as default.

![img.png](/media/tidb-cloud/serverless-external-storage/azure-create-sas.png)
![img.png](/media/tidb-cloud/serverless-external-storage/azure-create-sas.png)

4. Click the **Generate SAS and connection string** button to generate the SAS token.

0 comments on commit 06341eb

Please sign in to comment.