diff --git a/markdown-pages/en/tidbcloud/master/media/tidb-cloud/serverless-external-storage/serverless-role-arn.png b/markdown-pages/en/tidbcloud/master/media/tidb-cloud/serverless-external-storage/serverless-role-arn.png new file mode 100644 index 00000000..509f1bf0 Binary files /dev/null and b/markdown-pages/en/tidbcloud/master/media/tidb-cloud/serverless-external-storage/serverless-role-arn.png differ diff --git a/markdown-pages/en/tidbcloud/master/tidb-cloud/config-s3-and-gcs-access.md b/markdown-pages/en/tidbcloud/master/tidb-cloud/config-s3-and-gcs-access.md new file mode 100644 index 00000000..a53802b2 --- /dev/null +++ b/markdown-pages/en/tidbcloud/master/tidb-cloud/config-s3-and-gcs-access.md @@ -0,0 +1,216 @@ +--- +title: Configure External Storage Access for TiDB Dedicated +summary: Learn how to configure Amazon Simple Storage Service (Amazon S3) access and Google Cloud Storage (GCS) access. +--- + +# Configure External Storage Access for TiDB Dedicated + +If your source data is stored in Amazon S3 or Google Cloud Storage (GCS) buckets, before importing or migrating the data to TiDB Cloud, you need to configure cross-account access to the buckets. This document describes how to do this. + +If you need to configure these external storages for TiDB Serverless, see [Configure External Storage Access for TiDB Serverless](/tidb-cloud/serverless-external-storage.md). + +## Configure Amazon S3 access + +To allow TiDB Cloud to access the source data in your Amazon S3 bucket, you need to configure the bucket access for TiDB Cloud. You can use either of the following methods to configure the bucket access: + +- Use an AWS access key: use the access key of an IAM user to access your Amazon S3 bucket. +- Use a Role ARN: use a Role ARN to access your Amazon S3 bucket. + + +
+ +Configure the bucket access for TiDB Cloud and get the Role ARN as follows: + +1. In the [TiDB Cloud console](https://tidbcloud.com/), get the TiDB Cloud account ID and external ID of the target TiDB cluster. + + 1. Navigate to the [**Clusters**](https://tidbcloud.com/console/clusters) page of your project. + + > **Tip:** + > + > If you have multiple projects, you can click in the lower-left corner and switch to another project. + + 2. Click the name of your target cluster to go to its overview page, and then click **Import** in the left navigation pane. + + 3. On the **Import** page, click **Import Data** in the upper-right corner and select **From S3**. + + 4. On the **Import from S3** page, click **Guide for getting the required Role ARN** to get the TiDB Cloud Account ID and TiDB Cloud External ID. Take a note of these IDs for later use. + +2. In the AWS Management Console, create a managed policy for your Amazon S3 bucket. + + 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). + 2. In the **Buckets** list, choose the name of your bucket with the source data, and then click **Copy ARN** to get your S3 bucket ARN (for example, `arn:aws:s3:::tidb-cloud-source-data`). Take a note of the bucket ARN for later use. + + ![Copy bucket ARN](/media/tidb-cloud/copy-bucket-arn.png) + + 3. Open the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/), click **Policies** in the navigation pane on the left, and then click **Create Policy**. + + ![Create a policy](/media/tidb-cloud/aws-create-policy.png) + + 4. On the **Create policy** page, click the **JSON** tab. + 5. Copy the following access policy template and paste it to the policy text field. + + ```json + { + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "VisualEditor0", + "Effect": "Allow", + "Action": [ + "s3:GetObject", + "s3:GetObjectVersion" + ], + "Resource": "//*" + }, + { + "Sid": "VisualEditor1", + "Effect": "Allow", + "Action": [ + "s3:ListBucket", + "s3:GetBucketLocation" + ], + "Resource": "" + } + ] + } + ``` + + In the policy text field, update the following configurations to your own values. + + - `"Resource": "//*"` + + For example, if your source data is stored in the root directory of the `tidb-cloud-source-data` bucket, use `"Resource": "arn:aws:s3:::tidb-cloud-source-data/*"`. If your source data is stored in the `mydata` directory of the bucket, use `"Resource": "arn:aws:s3:::tidb-cloud-source-data/mydata/*"`. Make sure that `/*` is added to the end of the directory so TiDB Cloud can access all files in this directory. + + - `"Resource": ""` + + For example, `"Resource": "arn:aws:s3:::tidb-cloud-source-data"`. + + - If you have enabled AWS Key Management Service key (SSE-KMS) with customer-managed key encryption, make sure the following configuration is included in the policy. `"arn:aws:kms:ap-northeast-1:105880447796:key/c3046e91-fdfc-4f3a-acff-00597dd3801f"` is a sample KMS key of the bucket. + + ``` + { + "Sid": "AllowKMSkey", + "Effect": "Allow", + "Action": [ + "kms:Decrypt" + ], + "Resource": "arn:aws:kms:ap-northeast-1:105880447796:key/c3046e91-fdfc-4f3a-acff-00597dd3801f" + } + ``` + + If the objects in your bucket have been copied from another encrypted bucket, the KMS key value needs to include the keys of both buckets. For example, `"Resource": ["arn:aws:kms:ap-northeast-1:105880447796:key/c3046e91-fdfc-4f3a-acff-00597dd3801f","arn:aws:kms:ap-northeast-1:495580073302:key/0d7926a7-6ecc-4bf7-a9c1-a38f0faec0cd"]`. + + 6. Click **Next: Tags**, add a tag of the policy (optional), and then click **Next:Review**. + + 7. Set a policy name, and then click **Create policy**. + +3. In the AWS Management Console, create an access role for TiDB Cloud and get the role ARN. + + 1. In the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/), click **Roles** in the navigation pane on the left, and then click **Create role**. + + ![Create a role](/media/tidb-cloud/aws-create-role.png) + + 2. To create a role, fill in the following information: + + - Under **Trusted entity type**, select **AWS account**. + - Under **An AWS account**, select **Another AWS account**, and then paste the TiDB Cloud account ID to the **Account ID** field. + - Under **Options**, click **Require external ID (Best practice when a third party will assume this role)**, and then paste the TiDB Cloud External ID to the **External ID** field. If the role is created without "Require external ID", once the configuration is done for one TiDB cluster in a project, all TiDB clusters in that project can use the same Role ARN to access your Amazon S3 bucket. If the role is created with the account ID and external ID, only the corresponding TiDB cluster can access the bucket. + + 3. Click **Next** to open the policy list, choose the policy you just created, and then click **Next**. + 4. Under **Role details**, set a name for the role, and then click **Create role** in the lower-right corner. After the role is created, the list of roles is displayed. + 5. In the list of roles, click the name of the role that you just created to go to its summary page, and then copy the role ARN. + + ![Copy AWS role ARN](/media/tidb-cloud/aws-role-arn.png) + +4. In the TiDB Cloud console, go to the **Data Import** page where you get the TiDB Cloud account ID and external ID, and then paste the role ARN to the **Role ARN** field. + +
+
+ +It is recommended that you use an IAM user (instead of the AWS account root user) to create an access key. + +Take the following steps to configure an access key: + +1. Create an IAM user with the following policies: + + - `AmazonS3ReadOnlyAccess` + - [`CreateOwnAccessKeys` (required) and `ManageOwnAccessKeys` (optional)](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#access-keys_required-permissions) + + It is recommended that these policies only work for your bucket that stores the source data. + + For more information, see [Creating an IAM user](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html#id_users_create_console). + +2. Use your AWS account ID or account alias, and your IAM user name and password to sign in to [the IAM console](https://console.aws.amazon.com/iam). + +3. Create an access key. For more details, see [Creating an access key for an IAM user](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_CreateAccessKey). + +> **Note:** +> +> TiDB Cloud does not store your access keys. It is recommended that you [delete the access key](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_CreateAccessKey) after the import is complete. + +
+
+ +## Configure GCS access + +To allow TiDB Cloud to access the source data in your GCS bucket, you need to configure the GCS access for the bucket. Once the configuration is done for one TiDB cluster in a project, all TiDB clusters in that project can access the GCS bucket. + +1. In the TiDB Cloud console, get the Google Cloud Service Account ID of the target TiDB cluster. + + 1. Navigate to the [**Clusters**](https://tidbcloud.com/console/clusters) page of your project. + + > **Tip:** + > + > If you have multiple projects, you can click in the lower-left corner and switch to another project. + + 2. Click the name of your target cluster to go to its overview page, and then click **Import** in the left navigation pane. + + 3. Click **Import Data** in the upper-right corner, click **Show Google Cloud Service Account ID**, and then copy the Service Account ID for later use. + +2. In the Google Cloud console, create an IAM role for your GCS bucket. + + 1. Sign in to the [Google Cloud console](https://console.cloud.google.com/). + 2. Go to the [Roles](https://console.cloud.google.com/iam-admin/roles) page, and then click **CREATE ROLE**. + + ![Create a role](/media/tidb-cloud/gcp-create-role.png) + + 3. Enter a name, description, ID, and role launch stage for the role. The role name cannot be changed after the role is created. + 4. Click **ADD PERMISSIONS**. + 5. Add the following read-only permissions to the role, and then click **Add**. + + - storage.buckets.get + - storage.objects.get + - storage.objects.list + + You can copy a permission name to the **Enter property name or value** field as a filter query, and choose the name in the filter result. To add the three permissions, you can use **OR** between the permission names. + + ![Add permissions](/media/tidb-cloud/gcp-add-permissions.png) + +3. Go to the [Bucket](https://console.cloud.google.com/storage/browser) page, and click the name of the GCS bucket you want TiDB Cloud to access. + +4. On the **Bucket details** page, click the **PERMISSIONS** tab, and then click **GRANT ACCESS**. + + ![Grant Access to the bucket ](/media/tidb-cloud/gcp-bucket-permissions.png) + +5. Fill in the following information to grant access to your bucket, and then click **SAVE**. + + - In the **New Principals** field, paste the Google Cloud Service Account ID of the target TiDB cluster. + - In the **Select a role** drop-down list, type the name of the IAM role you just created, and then choose the name from the filter result. + + > **Note:** + > + > To remove the access to TiDB Cloud, you can simply remove the access that you have granted. + +6. On the **Bucket details** page, click the **OBJECTS** tab. + + If you want to copy a file's gsutil URI, select the file, click **Open object overflow menu**, and then click **Copy gsutil URI**. + + ![Get bucket URI](/media/tidb-cloud/gcp-bucket-uri01.png) + + If you want to use a folder's gsutil URI, open the folder, and then click the copy button following the folder name to copy the folder name. After that, you need to add `gs://` to the beginning and `/` to the end of the name to get a correct URI of the folder. + + For example, if the folder name is `tidb-cloud-source-data`, you need to use `gs://tidb-cloud-source-data/` as the URI. + + ![Get bucket URI](/media/tidb-cloud/gcp-bucket-uri02.png) + +7. In the TiDB Cloud console, go to the **Data Import** page where you get the Google Cloud Service Account ID, and then paste the GCS bucket gsutil URI to the **Bucket gsutil URI** field. For example, paste `gs://tidb-cloud-source-data/`. diff --git a/markdown-pages/en/tidbcloud/master/tidb-cloud/serverless-export.md b/markdown-pages/en/tidbcloud/master/tidb-cloud/serverless-export.md index a0e62719..0f981ff7 100644 --- a/markdown-pages/en/tidbcloud/master/tidb-cloud/serverless-export.md +++ b/markdown-pages/en/tidbcloud/master/tidb-cloud/serverless-export.md @@ -19,9 +19,9 @@ You can export data to: - Local file - External storage - - [Amazon S3](https://aws.amazon.com/s3/) - - [Google Cloud Storage](https://cloud.google.com/storage) - - [Azure Blob Storage](https://azure.microsoft.com/en-us/services/storage/blobs/) + - [Amazon S3](https://aws.amazon.com/s3/) + - [Google Cloud Storage](https://cloud.google.com/storage) + - [Azure Blob Storage](https://azure.microsoft.com/en-us/services/storage/blobs/) > **Note:** > @@ -29,9 +29,9 @@ You can export data to: ### Local file -Exporting data to local file has the following limitations: +Exporting data to a local file has the following limitations: -- You can only download the exported data with the [TiDB Cloud CLI](/tidb-cloud/cli-reference.md), download on the console is not supported. +- You can only download the exported data with the [TiDB Cloud CLI](/tidb-cloud/cli-reference.md). Downloading on the TiDB Cloud console is not supported. - Exported data is saved in the stashing area and will expire after two days. You need to download the exported data in time. - If the storage space of stashing area is full, you will not be able to export data to the local file. @@ -41,10 +41,10 @@ To export data to Amazon S3, you need to provide the following information: - uri: `s3:///` - one of the following access methods: - - [access key](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html): make sure the access key has the `s3:PutObject` and `s3:ListBucket` permissions. - - [role arn](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference-arns.html): make sure the role arn has the `s3:PutObject` and `s3:ListBucket` permissions. + - [An access key](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html): make sure the access key has the `s3:PutObject` and `s3:ListBucket` permissions. + - [A role ARN](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference-arns.html): make sure the role ARN has the `s3:PutObject` and `s3:ListBucket` permissions. -See [Configure TiDB Cloud Serverless external storage access](/tidb-cloud/serverless-external-storage.md#configure-amazon-s3-access) to learn how to configure the Amazon S3 access. +See [Configure External Storage Access for TiDB Serverless](/tidb-cloud/serverless-external-storage.md#configure-amazon-s3-access) to learn how to configure the Amazon S3 access. ### Google Cloud Storage @@ -53,7 +53,7 @@ To export data to Google Cloud Storage, you need to provide the following inform - uri: `gs:///` - access method: a **base64 encoded** [service account key](https://cloud.google.com/iam/docs/creating-managing-service-account-keys) for your bucket. Make sure the service account key has the `storage.objects.create` permission. -See [Configure TiDB Cloud Serverless external storage access](/tidb-cloud/serverless-external-storage.md#configure-gcs-access) to learn how to configure the Google Cloud Storage access. +See [Configure External Storage Access for TiDB Serverless](/tidb-cloud/serverless-external-storage.md#configure-gcs-access) to learn how to configure the Google Cloud Storage access. > **Note:** > @@ -66,7 +66,7 @@ To export data to Azure Blob Storage, you need to provide the following informat - uri: `azure://.blob.core.windows.net//` - access method: a [shared access signature (SAS) token](https://docs.microsoft.com/en-us/azure/storage/common/storage-sas-overview) for your Azure Blob Storage container. Make sure the SAS token has the `Read` and `Write` permissions on the `Container` and `Object` resources. -See [Configure TiDB Cloud Serverless external storage access](/tidb-cloud/serverless-external-storage.md#configure-azure-blob-access) to learn how to configure the Azure Blob Storage access. +See [Configure External Storage Access for TiDB Serverless](/tidb-cloud/serverless-external-storage.md#configure-azure-blob-access) to learn how to configure the Azure Blob Storage access. > **Note:** > @@ -85,11 +85,11 @@ You can export data in the following formats: - `SQL`: export data in SQL format. - `CSV`: export data in CSV format, you can specify the following options: - - `delimiter`: specify the delimiter used in the exported data. The default delimiter is `"`. - - `separator`: specify the character used to separate fields in the exported data. The default separator is `,`. - - `header`: specify whether to include a header row in the exported data. The default value is `true`. - - `null-value`: specify the string that represents a NULL value in the exported data. The default value is `\N`. -- `Parquet`: export data in Parquet format.(Only supported in TiDB Cloud CLI now) + - `delimiter`: specify the delimiter used in the exported data. The default delimiter is `"`. + - `separator`: specify the character used to separate fields in the exported data. The default separator is `,`. + - `header`: specify whether to include a header row in the exported data. The default value is `true`. + - `null-value`: specify the string that represents a NULL value in the exported data. The default value is `\N`. +- `Parquet`: export data in Parquet format. It is only supported in TiDB Cloud CLI now. The schema and data are exported according to the following naming conventions: @@ -101,68 +101,63 @@ The schema and data are exported according to the following naming conventions: ### Data compression -You can compress the exported csv and sql data using the following algorithms: +You can compress the exported CSV and SQL data using the following algorithms: - `gzip` (default): compress the exported data with gzip. - `snappy`: compress the exported data with snappy. - `zstd`: compress the exported data with zstd. - `none`: do not compress the exported data. -You can compress the exported parquet data using the following algorithms: -- `zstd`(default): compress the parquet file with zstd. -- `gzip`: compress the parquet file with gzip. -- `snappy`: compress the parquet file with snappy. -- `none`: do not compress the parquet file. +You can compress the exported Parquet data using the following algorithms: + +- `zstd `(default): compress the Parquet file with zstd. +- `gzip`: compress the Parquet file with gzip. +- `snappy`: compress the Parquet file with snappy. +- `none`: do not compress the Parquet file. ## Examples -### Export data to local file +### Export data to a local file
-1. Open the **Import** page for your target cluster. - - 1. Log in to the [TiDB Cloud console](https://tidbcloud.com/) and navigate to the [**Clusters**](https://tidbcloud.com/console/clusters) page of your project. - - > **Tip:** - > - > If you have multiple projects, you can click in the lower-left corner and switch to another project. - - 2. Click the name of your target cluster to go to its overview page, and then click **Import** in the left navigation pane. +1. Log in to the [TiDB Cloud console](https://tidbcloud.com/) and navigate to the [**Clusters**](https://tidbcloud.com/console/clusters) page of your project. -2. Configure the export information. On the **Import** page: + > **Tip:** + > + > If you have multiple projects, you can click in the lower-left corner and switch to another project. - - Click **Export Data to** in the upper-right corner, then choose **Local File** from the drop-down list. +2. Click the name of your target cluster to go to its overview page, and then click **Import & Export** in the left navigation pane. - Fill in the following parameters: +3. On the **Import & Export** page, click **Export Data to** in the upper-right corner, then choose **Local File** from the drop-down list. Fill in the following parameters: - - **Task Name**: enter a name for the export task, default is `SNAPSHOT_{snapshot_time}`. + - **Task Name**: enter a name for the export task. The default is `SNAPSHOT_{snapshot_time}`. - **Exported data**: choose the databases and tables you want to export. - - **Data format**: choose one of the **SQL File** and **CSV**. - - **Compression**: choose one of the **Gzip**, **Snappy**, **Zstd**, and **None**. + - **Data format**: choose **SQL File** or **CSV**. + - **Compression**: choose **Gzip**, **Snappy**, **Zstd**, or **None**. -3. Click **Export**. +4. Click **Export**. -4. After the export task is successful, you can download the exported data to your local file with TiDB Cloud CLI. +5. After the export task is successful, you can download the exported data to your local file with TiDB Cloud CLI.
-1. Create an export task, cli will export the data to local file by default: +1. Create an export task. TiDB Cloud CLI will export the data to local file by default: - ```shell - ticloud serverless export create -c --filter "database.table" - ``` + ```shell + ticloud serverless export create -c --filter "database.table" + ``` - You will get an export ID from the output. + You will get an export ID from the output. 2. After the export task is successful, download the exported data to your local file: - ```shell - ticloud serverless export download -c -e - ``` + ```shell + ticloud serverless export download -c -e + ```
@@ -172,34 +167,28 @@ You can compress the exported parquet data using the following algorithms:
-1. Open the **Import** page for your target cluster. - - 1. Log in to the [TiDB Cloud console](https://tidbcloud.com/) and navigate to the [**Clusters**](https://tidbcloud.com/console/clusters) page of your project. +1. Log in to the [TiDB Cloud console](https://tidbcloud.com/) and navigate to the [**Clusters**](https://tidbcloud.com/console/clusters) page of your project. > **Tip:** > > If you have multiple projects, you can click in the lower-left corner and switch to another project. - 2. Click the name of your target cluster to go to its overview page, and then click **Import** in the left navigation pane. - -2. Configure the export information. On the **Import** page: - - - Click **Export Data to** in the upper-right corner, then choose **Amazon S3** from the drop-down list. +2. Click the name of your target cluster to go to its overview page, and then click **Import & Export** in the left navigation pane. - Fill in the following parameters: +3. Configure the export information. On the **Import & Export** page, click **Export Data to** in the upper-right corner, then choose **Amazon S3** from the drop-down list. Fill in the following parameters: - - **Task Name**: enter a name for the export task, default is `SNAPSHOT_{snapshot_time}`. + - **Task Name**: enter a name for the export task. The default value is `SNAPSHOT_{snapshot_time}`. - **Exported data**: choose the databases and tables you want to export. - - **Data format**: choose one of the **SQL File** and **CSV**. - - **Compression**: choose one of the **Gzip**, **Snappy**, **Zstd**, and **None**. + - **Data format**: choose **SQL File** or **CSV**. + - **Compression**: choose **Gzip**, **Snappy**, **Zstd**, or **None**. - **File URI**: enter the URI of the Amazon S3 with the `s3:///` format. - **Bucket Access** - - **AWS Role Arn**: enter the ARN of the role that has the permission to access the bucket. - - **AWS Access Key ID**: enter the access key ID and access key secret that has the permission to access the bucket. + - **AWS Role ARN**: enter the role ARN that has the permission to access the bucket. + - **AWS Access Key ID**: enter the access key ID and access key secret that have the permission to access the bucket. -3. Click **Export**. +4. Click **Export**. -4. After the export task is successful, you can download the exported data to your local file with TiDB Cloud CLI. +5. After the export task is successful, you can download the exported data to your local file with TiDB Cloud CLI.
@@ -209,16 +198,16 @@ You can compress the exported parquet data using the following algorithms: ticloud serverless export create -c --s3.uri --s3.access-key-id --s3.secret-access-key --filter "database.table" ``` -- s3.uri: The Amazon S3 URI with the `s3:///` format. -- s3.access-key-id: The access key ID of the user who has the permission to access the bucket. -- s3.secret-access-key: The access key secret of the user who has the permission to access the bucket. +- s3.uri: the Amazon S3 URI with the `s3:///` format. +- s3.access-key-id: the access key ID of the user who has the permission to access the bucket. +- s3.secret-access-key: the access key secret of the user who has the permission to access the bucket. ```shell ticloud serverless export create -c --s3.uri --s3.role-arn --filter "database.table" ``` -- s3.uri: The URI of the Amazon S3 bucket with the `s3:///` format. -- s3.role-arn: The ARN of the role that has the permission to access the bucket. +- s3.uri: the URI of the Amazon S3 bucket with the `s3:///` format. +- s3.role-arn: the ARN of the role that has the permission to access the bucket.
@@ -231,8 +220,8 @@ You can only export data to Google Cloud Storage using the TiDB Cloud CLI now. ticloud serverless export create -c --gcs.uri --gcs.service-account-key --filter "database.table" ``` -- gcs.uri: The URI of the Google Cloud Storage bucket with the `gs:///` format. -- gcs.service-account-key: The base64 encoded service account key. +- gcs.uri: the URI of the Google Cloud Storage bucket with the `gs:///` format. +- gcs.service-account-key: the base64 encoded service account key. ### Export data to Azure Blob Storage @@ -242,29 +231,27 @@ You can only export data to Azure Blob Storage using the TiDB Cloud CLI now. ticloud serverless export create -c --azblob.uri --azblob.sas-token --filter "database.table" ``` -- azblob.uri: The URI of the Azure Blob Storage with the `azure://.blob.core.windows.net//` format. -- azblob.sas-token: The account SAS token of the Azure Blob Storage. +- azblob.uri: the URI of the Azure Blob Storage with the `azure://.blob.core.windows.net//` format. +- azblob.sas-token: the account SAS token of the Azure Blob Storage. ### Cancel an export task
-1. Open the **Import** page for your target cluster. +1. Open the **Import & Export** page for your target cluster, log in to the [TiDB Cloud console](https://tidbcloud.com/) and navigate to the [**Clusters**](https://tidbcloud.com/console/clusters) page of your project. - 1. Log in to the [TiDB Cloud console](https://tidbcloud.com/) and navigate to the [**Clusters**](https://tidbcloud.com/console/clusters) page of your project. + > **Tip:** + > + > If you have multiple projects, you can click in the lower-left corner and switch to another project. - > **Tip:** - > - > If you have multiple projects, you can click in the lower-left corner and switch to another project. +2. Click the name of your target cluster to go to its overview page, and then click **Import & Export** in the left navigation pane. - 2. Click the name of your target cluster to go to its overview page, and then click **Import** in the left navigation pane. +3. On the **Import & Export** page, click **Export** to view the export task list. -2. Cancel the export task. On the **Import** page: +4. Choose the export task you want to cancel, and then click **Action** button. - - Click **Export** to view the export task list. - - Choose the export task you want to cancel, and then click **Action** button. - - Choose **Cancel** in the drop-down list, note that you can only cancel the export task that is in the **Running** status. +5. Choose **Cancel** in the drop-down list. Note that you can only cancel the export task that is in the **Running** status.
diff --git a/markdown-pages/en/tidbcloud/master/tidb-cloud/serverless-external-storage.md b/markdown-pages/en/tidbcloud/master/tidb-cloud/serverless-external-storage.md index 8439161f..6c96c123 100644 --- a/markdown-pages/en/tidbcloud/master/tidb-cloud/serverless-external-storage.md +++ b/markdown-pages/en/tidbcloud/master/tidb-cloud/serverless-external-storage.md @@ -1,15 +1,17 @@ --- -title: Configure TiDB Cloud Serverless external storage access +title: Configure TiDB Serverless External Storage Access summary: Learn how to configure Amazon Simple Storage Service (Amazon S3) access, Google Cloud Storage (GCS) access and Azure Blob Storage access. --- -# Configure TiDB Cloud Serverless external storage access +# Configure External Storage Access for TiDB Serverless -If you want import data from or export data to external storage in TiDB Cloud Serverless, you need to configure cross-account access. This document describes how to configure access to external storage, such as Amazon Simple Storage Service (Amazon S3), Google Cloud Storage (GCS) and Azure Blob Storage for TiDB Cloud Serverless. +If you want import data from or export data to external storage in TiDB Serverless, you need to configure cross-account access. This document describes how to configure access to an external storage, including Amazon Simple Storage Service (Amazon S3), Google Cloud Storage (GCS) and Azure Blob Storage for TiDB Serverless. + +If you need to configure these external storages for TiDB Dedicated, see [Configure Amazon S3 Access and GCS Access for TiDB Dedicated](/tidb-cloud/config-s3-and-gcs-access.md). ## Configure Amazon S3 access -To allow TiDB Cloud Serverless to access your Amazon S3 bucket, you need to configure the bucket access for TiDB Cloud Serverless. You can use either of the following methods to configure the bucket access: +To allow TiDB Serverless to access your Amazon S3 bucket, you need to configure the bucket access for TiDB Serverless. You can use either of the following methods to configure the bucket access: - Use an AWS access key: use the access key of an IAM user to access your Amazon S3 bucket. - Use a Role ARN: use a Role ARN to access your Amazon S3 bucket. @@ -17,7 +19,139 @@ To allow TiDB Cloud Serverless to access your Amazon S3 bucket, you need to conf
+It is recommended that you use AWS CloudFormation to create a role ARN. + +Take the following steps to create one: + +1. Open the **Import & Export** page for your target cluster. + + 1. Log in to the [TiDB Cloud console](https://tidbcloud.com/) and navigate to the [**Clusters**](https://tidbcloud.com/console/clusters) page of your project. + + 2. Click the name of your target cluster to go to its overview page, and then click **Import & Export** in the left navigation pane. + +2. Open the `Add New ARN` pop-up window. + + 1. Click **Import from S3** or **Export to Amazon S3** according to your needs. + + 2. Fill in the `Folder URI` or `File URI` according to the instructions. + + 3. Choose **AWS Role ARN** and click **Click here to create new one with AWS CloudFormation** to open the `Add New ARN` pop-up window. + +3. Create a role ARN with AWS CloudFormation Template. + + 1. Click **AWS Console with CloudFormation Template** in the `Add New ARN` pop-up window. + + 2. Log in to the AWS and you will be redirected to the AWS CloudFormation Template page. + + 3. Fill in the `Role Name`. + + 4. Acknowledge to create a new role and click **Create stack** to create the role ARN. + + 5. After the CloudFormation stack is executed, you can find the Role ARN value in the `outputs` tab. + + ![img.png](/media/tidb-cloud/serverless-external-storage/serverless-role-arn.png) + +
+ +
+ +If you have any trouble creating a role ARN with AWS CloudFormation, you can create one manually. + +Take the following steps to create one: + +1. In the `Add New ARN` pop-up window described in the previous section, click **Having trouble? Create Role ARN manually. You will get the `TiDB Cloud Account ID` and `TiDB Cloud External ID`. + +2. In the AWS Management Console, create a managed policy for your Amazon S3 bucket. + + 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). + 2. In the **Buckets** list, choose the name of your bucket with the source data, and then click **Copy ARN** to get your S3 bucket ARN (for example, `arn:aws:s3:::tidb-cloud-source-data`). Take a note of the bucket ARN for later use. + + ![Copy bucket ARN](/media/tidb-cloud/copy-bucket-arn.png) + + 3. Open the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/), click **Policies** in the navigation pane on the left, and then click **Create Policy**. + + ![Create a policy](/media/tidb-cloud/aws-create-policy.png) + + 4. On the **Create policy** page, click the **JSON** tab. + 5. Copy the following access policy template and paste it to the policy text field. + + ```json + { + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "VisualEditor0", + "Effect": "Allow", + "Action": [ + "s3:GetObject", + "s3:GetObjectVersion", + "s3:PutObject" + ], + "Resource": "//*" + }, + { + "Sid": "VisualEditor1", + "Effect": "Allow", + "Action": [ + "s3:ListBucket", + "s3:GetBucketLocation" + ], + "Resource": "" + } + ] + } + ``` + + In the policy text field, update the following configurations to your own values. + + - `"Resource": "//*"` + + For example, if your source data is stored in the root directory of the `tidb-cloud-source-data` bucket, use `"Resource": "arn:aws:s3:::tidb-cloud-source-data/*"`. If your source data is stored in the `mydata` directory of the bucket, use `"Resource": "arn:aws:s3:::tidb-cloud-source-data/mydata/*"`. Make sure that `/*` is added to the end of the directory so TiDB Cloud can access all files in this directory. + + - `"Resource": ""` + + For example, `"Resource": "arn:aws:s3:::tidb-cloud-source-data"`. + + - If you have enabled AWS Key Management Service key (SSE-KMS) with customer-managed key encryption, make sure the following configuration is included in the policy. `"arn:aws:kms:ap-northeast-1:105880447796:key/c3046e91-fdfc-4f3a-acff-00597dd3801f"` is a sample KMS key of the bucket. + + ``` + { + "Sid": "AllowKMSkey", + "Effect": "Allow", + "Action": [ + "kms:Decrypt" + ], + "Resource": "arn:aws:kms:ap-northeast-1:105880447796:key/c3046e91-fdfc-4f3a-acff-00597dd3801f" + } + ``` + + If the objects in your bucket have been copied from another encrypted bucket, the KMS key value needs to include the keys of both buckets. For example, `"Resource": ["arn:aws:kms:ap-northeast-1:105880447796:key/c3046e91-fdfc-4f3a-acff-00597dd3801f","arn:aws:kms:ap-northeast-1:495580073302:key/0d7926a7-6ecc-4bf7-a9c1-a38f0faec0cd"]`. + + 6. Click **Next: Tags**, add a tag of the policy (optional), and then click **Next:Review**. + + 7. Set a policy name, and then click **Create policy**. + +3. In the AWS Management Console, create an access role for TiDB Cloud and get the role ARN. + + 1. In the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/), click **Roles** in the navigation pane on the left, and then click **Create role**. + + ![Create a role](/media/tidb-cloud/aws-create-role.png) + + 2. To create a role, fill in the following information: + + - Under **Trusted entity type**, select **AWS account**. + - Under **An AWS account**, select **Another AWS account**, and then paste the TiDB Cloud account ID to the **Account ID** field. + - Under **Options**, click **Require external ID (Best practice when a third party will assume this role)**, and then paste the TiDB Cloud External ID to the **External ID** field. If the role is created without "Require external ID", once the configuration is done for one TiDB cluster in a project, all TiDB clusters in that project can use the same Role ARN to access your Amazon S3 bucket. If the role is created with the account ID and external ID, only the corresponding TiDB cluster can access the bucket. + + 3. Click **Next** to open the policy list, choose the policy you just created, and then click **Next**. + 4. Under **Role details**, set a name for the role, and then click **Create role** in the lower-right corner. After the role is created, the list of roles is displayed. + 5. In the list of roles, click the name of the role that you just created to go to its summary page, and then you can get the role ARN. + + ![Copy AWS role ARN](/media/tidb-cloud/aws-role-arn.png) + +
+
It is recommended that you use an IAM user (instead of the AWS account root user) to create an access key. @@ -39,15 +173,15 @@ Take the following steps to configure an access key: ## Configure GCS access -To allow TiDB Cloud Serverless to access your GCS bucket, you need to configure the GCS access for the bucket. You can use service account key to configure the bucket access: +To allow TiDB Serverless to access your GCS bucket, you need to configure the GCS access for the bucket. You can use service account key to configure the bucket access: Take the following steps to configure a service account key: 1. Create a service account with needed permission in the Google Cloud [service account page](https://console.cloud.google.com/iam-admin/serviceaccounts). For more information, see [Creating a service account](https://cloud.google.com/iam/docs/creating-managing-service-accounts). 1. Enter a service account name. - 2. Optional: Enter a description of the service account. + 2. Enter a description of the service account (Optional). 3. Click **CREATE AND CONTINUE** to create the service account and continue the next step. - 4. In the `Grant this service account access to project`, choose the [IAM roles](https://cloud.google.com/iam/docs/understanding-roles) with needed permission. For example, TiDB Cloud Serverless export needs a role with `storage.objects.create` permission. + 4. In the `Grant this service account access to project`, choose the [IAM roles](https://cloud.google.com/iam/docs/understanding-roles) with needed permission. For example, TiDB Serverless export needs a role with `storage.objects.create` permission. 5. Click **Continue** to the next step. 6. Optional: In the `Grant users access to this service account`, choose members that need to [attach the service account to other resources](https://cloud.google.com/iam/docs/attach-service-accounts). 7. Click **Done** to finish creating the service account. @@ -62,27 +196,28 @@ Take the following steps to configure a service account key: 4. Open the downloaded JSON file and encode the content with base64. For example, you can use the following command to encode the content in the macOS terminal: -```bash -base64 -i gcp-xxx.json -``` + ```bash + base64 -i gcp-xxx.json + ``` ## Configure Azure Blob access -To allow TiDB Cloud Serverless to access your Azure Blob container, you need to configure the Azure Blob access for the container. You can use service SAS token to configure the container access: +To allow TiDB Serverless to access your Azure Blob container, you need to configure the Azure Blob access for the container. You can use service SAS token to configure the container access: Take the following steps to configure a service SAS token: -1. Click your storage account where the container belong to in the [Azure storage account](https://portal.azure.com/#browse/Microsoft.Storage%2FStorageAccounts) page. +1. Click your storage account where the container belongs to in the [Azure storage account](https://portal.azure.com/#browse/Microsoft.Storage%2FStorageAccounts) page. + 2. In your `storage account` page, click the **Security+network** and then click the **Shared access signature**. ![img.png](/media/tidb-cloud/serverless-external-storage/azure-sas.png) -3. In the `Shared access signature` page, create a service SAS token with needed permission. For more information, see [Create a service SAS token](https://docs.microsoft.com/en-us/azure/storage/common/storage-sas-overview). - 1. Choose the `Blob` service under the `Allowed services` section. - 2. Choose the `Container` and `Object` under the `Allowed Resource types` section. - 3. Choose the permission under the `Allowed permissions` section as you needed. For example: TiDB Cloud Serverless exports needs `Read` and `Write` permission. - 4. Adjust the `Start and expiry date/time` as you needed. - 5. You can keep other settings as default. +3. In the `Shared access signature` page, create a service SAS token with needed permissions as follows. For more information, see [Create a service SAS token](https://docs.microsoft.com/en-us/azure/storage/common/storage-sas-overview). + 1. Choose the `Blob` service under the `Allowed services` section. + 2. Choose the `Container` and `Object` under the `Allowed Resource types` section. + 3. Choose the permission under the `Allowed permissions` section as you needed. For example: TiDB Serverless exports needs `Read` and `Write` permission. + 4. Adjust the `Start and expiry date/time` as you needed. + 5. You can keep other settings as default. ![img.png](/media/tidb-cloud/serverless-external-storage/azure-create-sas.png)