You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: tidb-cloud/serverless-external-storage.md
+97-95Lines changed: 97 additions & 95 deletions
Original file line number
Diff line number
Diff line change
@@ -5,13 +5,13 @@ summary: Learn how to configure Amazon Simple Storage Service (Amazon S3) access
5
5
6
6
# Configure External Storage Access for TiDB Serverless
7
7
8
-
If you want import data from or export data to external storage in TiDB Serverless, you need to configure cross-account access. This document describes how to configure access to an external storage, including Amazon Simple Storage Service (Amazon S3), Google Cloud Storage (GCS) and Azure Blob Storage for TiDB Serverless.
8
+
If you want import data from or export data to an external storage in TiDB Serverless, you need to configure cross-account access. This document describes how to configure access to an external storage, including Amazon Simple Storage Service (Amazon S3), Google Cloud Storage (GCS) and Azure Blob Storage for TiDB Serverless.
9
9
10
10
If you need to configure these external storages for TiDB Dedicated, see [Configure Amazon S3 Access and GCS Access for TiDB Dedicated](/tidb-cloud/config-s3-and-gcs-access.md).
11
11
12
12
## Configure Amazon S3 access
13
13
14
-
To allow TiDB Serverless to access your Amazon S3 bucket, you need to configure the bucket access for TiDB Serverless. You can use either of the following methods to configure the bucket access:
14
+
To allow a iDB Serverless cluster to access your Amazon S3 bucket, you need to configure the bucket access for TiDB Serverless. You can use either of the following methods to configure the bucket access:
15
15
16
16
- Use an AWS access key: use the access key of an IAM user to access your Amazon S3 bucket.
17
17
- Use a Role ARN: use a Role ARN to access your Amazon S3 bucket.
@@ -25,31 +25,31 @@ Take the following steps to create one:
25
25
26
26
1. Open the **Import & Export** page for your target cluster.
27
27
28
-
1. Log in to the [TiDB Cloud console](https://tidbcloud.com/) and navigate to the [**Clusters**](https://tidbcloud.com/console/clusters) page of your project.
28
+
1. Log in to the [TiDB Cloud console](https://tidbcloud.com/) and navigate to the [**Clusters**](https://tidbcloud.com/console/clusters) page of your project.
29
29
30
-
2. Click the name of your target cluster to go to its overview page, and then click **Import & Export** in the left navigation pane.
30
+
2. Click the name of your target cluster to go to its overview page, and then click **Import & Export** in the left navigation pane.
31
31
32
32
2. Open the `Add New ARN` pop-up window.
33
33
34
-
1. Click **Import from S3** or **Export to Amazon S3** according to your needs.
34
+
1. Click **Import from S3** or **Export to Amazon S3** according to your needs.
35
35
36
-
2. Fill in the `Folder URI` or `File URI` according to the instructions.
36
+
2. Fill in the `Folder URI` or `File URI` according to the instructions.
37
37
38
-
3. Choose **AWS Role ARN** and click **Click here to create new one with AWS CloudFormation** to open the **Add New ARN** pop-up window.
38
+
3. Choose **AWS Role ARN** and click **Click here to create new one with AWS CloudFormation** to open the **Add New ARN** pop-up window.
39
39
40
40
3. Create a role ARN with AWS CloudFormation Template.
41
41
42
-
1. Click **AWS Console with CloudFormation Template** in the **Add New ARN** pop-up window.
42
+
1. Click **AWS Console with CloudFormation Template** in the **Add New ARN** pop-up window.
43
43
44
-
2. Log in to the AWS and you will be redirected to the AWS CloudFormation Template page.
44
+
2. Log in to the AWS and you will be redirected to the AWS CloudFormation Template page.
45
45
46
-
3. Fill in the **Role Name**.
46
+
3. Fill in the **Role Name**.
47
47
48
-
4. Acknowledge to create a new role and click **Create stack** to create the role ARN.
48
+
4. Acknowledge to create a new role and click **Create stack** to create the role ARN.
49
49
50
-
5. After the CloudFormation stack is executed, you can find the Role ARN value in the **Outputs** tab.
50
+
5. After the CloudFormation stack is executed, you can find the Role ARN value in the **Outputs** tab.
@@ -64,95 +64,95 @@ Take the following steps to create one:
64
64
65
65
2. In the AWS Management Console, create a managed policy for your Amazon S3 bucket.
66
66
67
-
1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/).
67
+
1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/).
68
68
69
-
2. In the **Buckets** list, choose the name of your bucket with the source data, and then click **Copy ARN** to get your S3 bucket ARN (for example, `arn:aws:s3:::tidb-cloud-source-data`). Take a note of the bucket ARN for later use.
69
+
2. In the **Buckets** list, choose the name of your bucket with the source data, and then click **Copy ARN** to get your S3 bucket ARN (for example, `arn:aws:s3:::tidb-cloud-source-data`). Take a note of the bucket ARN for later use.
3. Open the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/), click **Policies** in the navigation pane on the left, and then click **Create Policy**.
73
+
3. Open the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/), click **Policies** in the navigation pane on the left, and then click **Create Policy**.
74
74
75
-
75
+
76
76
77
-
4. On the **Create policy** page, click the **JSON** tab.
77
+
4. On the **Create policy** page, click the **JSON** tab.
78
78
79
-
5. Copy the following access policy template and paste it to the policy text field.
80
-
81
-
```json
82
-
{
83
-
"Version": "2012-10-17",
84
-
"Statement": [
85
-
{
86
-
"Sid": "VisualEditor0",
87
-
"Effect": "Allow",
88
-
"Action": [
89
-
"s3:GetObject",
90
-
"s3:GetObjectVersion",
91
-
"s3:PutObject"
92
-
],
93
-
"Resource": "<Your S3 bucket ARN>/<Directory of your source data>/*"
94
-
},
95
-
{
96
-
"Sid": "VisualEditor1",
97
-
"Effect": "Allow",
98
-
"Action": [
99
-
"s3:ListBucket",
100
-
"s3:GetBucketLocation"
101
-
],
102
-
"Resource": "<Your S3 bucket ARN>"
103
-
}
104
-
]
105
-
}
106
-
```
107
-
108
-
In the policy text field, update the following configurations to your own values.
109
-
110
-
- `"Resource": "<Your S3 bucket ARN>/<Directory of the source data>/*"`
111
-
112
-
For example, if your source data is stored in the root directory of the `tidb-cloud-source-data` bucket, use `"Resource": "arn:aws:s3:::tidb-cloud-source-data/*"`. If your source data is stored in the `mydata` directory of the bucket, use `"Resource": "arn:aws:s3:::tidb-cloud-source-data/mydata/*"`. Make sure that `/*` is added to the end of the directory so TiDB Cloud can access all files in this directory.
113
-
114
-
- `"Resource": "<Your S3 bucket ARN>"`
115
-
116
-
For example, `"Resource": "arn:aws:s3:::tidb-cloud-source-data"`.
117
-
118
-
- If you have enabled AWS Key Management Service key (SSE-KMS) with customer-managed key encryption, make sure the following configuration is included in the policy. `"arn:aws:kms:ap-northeast-1:105880447796:key/c3046e91-fdfc-4f3a-acff-00597dd3801f"` is a sample KMS key of the bucket.
If the objects in your bucket have been copied from another encrypted bucket, the KMS key value needs to include the keys of both buckets. For example, `"Resource": ["arn:aws:kms:ap-northeast-1:105880447796:key/c3046e91-fdfc-4f3a-acff-00597dd3801f","arn:aws:kms:ap-northeast-1:495580073302:key/0d7926a7-6ecc-4bf7-a9c1-a38f0faec0cd"]`.
132
-
133
-
6. Click **Next: Tags**, add a tag of the policy (optional), and then click **Next:Review**.
134
-
135
-
7. Set a policy name, and then click **Create policy**.
79
+
5. Copy the following access policy template and paste it to the policy text field.
80
+
81
+
```json
82
+
{
83
+
"Version": "2012-10-17",
84
+
"Statement": [
85
+
{
86
+
"Sid": "VisualEditor0",
87
+
"Effect": "Allow",
88
+
"Action": [
89
+
"s3:GetObject",
90
+
"s3:GetObjectVersion",
91
+
"s3:PutObject"
92
+
],
93
+
"Resource": "<Your S3 bucket ARN>/<Directory of your source data>/*"
94
+
},
95
+
{
96
+
"Sid": "VisualEditor1",
97
+
"Effect": "Allow",
98
+
"Action": [
99
+
"s3:ListBucket",
100
+
"s3:GetBucketLocation"
101
+
],
102
+
"Resource": "<Your S3 bucket ARN>"
103
+
}
104
+
]
105
+
}
106
+
```
107
+
108
+
In the policy text field, update the following configurations to your own values.
109
+
110
+
- `"Resource": "<Your S3 bucket ARN>/<Directory of the source data>/*"`
111
+
112
+
For example, if your source data is stored in the root directory of the `tidb-cloud-source-data` bucket, use `"Resource": "arn:aws:s3:::tidb-cloud-source-data/*"`. If your source data is stored in the `mydata` directory of the bucket, use `"Resource": "arn:aws:s3:::tidb-cloud-source-data/mydata/*"`. Make sure that `/*` is added to the end of the directory so TiDB Cloud can access all files in this directory.
113
+
114
+
- `"Resource": "<Your S3 bucket ARN>"`
115
+
116
+
For example, `"Resource": "arn:aws:s3:::tidb-cloud-source-data"`.
117
+
118
+
- If you have enabled AWS Key Management Service key (SSE-KMS) with customer-managed key encryption, make sure the following configuration is included in the policy. `"arn:aws:kms:ap-northeast-1:105880447796:key/c3046e91-fdfc-4f3a-acff-00597dd3801f"` is a sample KMS key of the bucket.
- If the objects in your bucket have been copied from another encrypted bucket, the KMS key value needs to include the keys of both buckets. For example, `"Resource": ["arn:aws:kms:ap-northeast-1:105880447796:key/c3046e91-fdfc-4f3a-acff-00597dd3801f","arn:aws:kms:ap-northeast-1:495580073302:key/0d7926a7-6ecc-4bf7-a9c1-a38f0faec0cd"]`.
132
+
133
+
6. Click **Next: Tags**, add a tag of the policy (optional), and then click **Next:Review**.
134
+
135
+
7. Set a policy name, and then click **Create policy**.
136
136
137
137
3. In the AWS Management Console, create an access role for TiDB Cloud and get the role ARN.
138
138
139
-
1. In the [IAM console](https://console.aws.amazon.com/iam/), click **Roles** in the navigation pane on the left, and then click **Create role**.
139
+
1. In the [IAM console](https://console.aws.amazon.com/iam/), click **Roles** in the navigation pane on the left, and then click **Create role**.
140
140
141
-
141
+
142
142
143
-
2. To create a role, fill in the following information:
143
+
2. To create a role, fill in the following information:
144
144
145
-
- Under **Trusted entity type**, select **AWS account**.
146
-
- Under **An AWS account**, select **Another AWS account**, and then paste the TiDB Cloud account ID to the **Account ID** field.
147
-
- Under **Options**, click **Require external ID (Best practice when a third party will assume this role)**, and then paste the TiDB Cloud External ID to the **External ID** field. If the role is created without "Require external ID", once the configuration is done for one TiDB cluster in a project, all TiDB clusters in that project can use the same Role ARN to access your Amazon S3 bucket. If the role is created with the account ID and external ID, only the corresponding TiDB cluster can access the bucket.
145
+
- Under **Trusted entity type**, select **AWS account**.
146
+
- Under **An AWS account**, select **Another AWS account**, and then paste the TiDB Cloud account ID to the **Account ID** field.
147
+
- Under **Options**, click **Require external ID (Best practice when a third party will assume this role)**, and then paste the TiDB Cloud External ID to the **External ID** field. If the role is created without "Require external ID", once the configuration is done for one TiDB cluster in a project, all TiDB clusters in that project can use the same Role ARN to access your Amazon S3 bucket. If the role is created with the account ID and external ID, only the corresponding TiDB cluster can access the bucket.
148
148
149
-
3. Click **Next** to open the policy list, choose the policy you just created, and then click **Next**.
149
+
3. Click **Next** to open the policy list, choose the policy you just created, and then click **Next**.
150
150
151
-
4. Under **Role details**, set a name for the role, and then click **Create role** in the lower-right corner. After the role is created, the list of roles is displayed.
151
+
4. Under **Role details**, set a name for the role, and then click **Create role** in the lower-right corner. After the role is created, the list of roles is displayed.
152
152
153
-
5. In the list of roles, click the name of the role that you just created to go to its summary page, and then you can get the role ARN.
153
+
5. In the list of roles, click the name of the role that you just created to go to its summary page, and then you can get the role ARN.
154
154
155
-
155
+
156
156
157
157
</div>
158
158
@@ -182,15 +182,16 @@ To allow TiDB Serverless to access your GCS bucket, you need to configure the GC
182
182
Take the following steps to configure a service account key:
183
183
184
184
1. Create a service account with needed permission in the Google Cloud [service account page](https://console.cloud.google.com/iam-admin/serviceaccounts). For more information, see [Creating a service account](https://cloud.google.com/iam/docs/creating-managing-service-accounts).
185
-
1. Enter a service account name.
186
-
2. Enter a description of the service account (Optional).
187
-
3. Click **CREATE AND CONTINUE** to create the service account and continue the next step.
188
-
4. In the `Grant this service account access to project`, choose the [IAM roles](https://cloud.google.com/iam/docs/understanding-roles) with needed permission. For example, TiDB Serverless export needs a role with `storage.objects.create` permission.
189
-
5. Click **Continue** to the next step.
190
-
6. Optional: In the `Grant users access to this service account`, choose members that need to [attach the service account to other resources](https://cloud.google.com/iam/docs/attach-service-accounts).
191
-
7. Click **Done** to finish creating the service account.
2. Enter a description of the service account (Optional).
188
+
3. Click **CREATE AND CONTINUE** to create the service account and continue the next step.
189
+
4. In the `Grant this service account access to project`, choose the [IAM roles](https://cloud.google.com/iam/docs/understanding-roles) with needed permission. For example, TiDB Serverless export needs a role with `storage.objects.create` permission.
190
+
5. Click **Continue** to the next step.
191
+
6. Optional: In the `Grant users access to this service account`, choose members that need to [attach the service account to other resources](https://cloud.google.com/iam/docs/attach-service-accounts).
192
+
7. Click **Done** to finish creating the service account.
3. In the `Shared access signature` page, create a service SAS token with needed permissions as follows. For more information, see [Create a service SAS token](https://docs.microsoft.com/en-us/azure/storage/common/storage-sas-overview).
221
+
220
222
1. Choose the `Blob` service under the `Allowed services` section.
221
223
2. Choose the `Container` and `Object` under the `Allowed Resource types` section.
222
224
3. Choose the permission under the `Allowed permissions` section as you needed. For example: TiDB Serverless exports needs `Read` and `Write` permission.
223
225
4. Adjust the `Start and expiry date/time` as you needed.
0 commit comments