-
Notifications
You must be signed in to change notification settings - Fork 2
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* 550 byovpc for AWS * update cloud overview * Update modules/get-started/pages/cluster-types/byoc/create-byoc-cluster-aws.adoc Co-authored-by: Joyce Fee <[email protected]> * Update modules/get-started/pages/whats-new-cloud.adoc Co-authored-by: Joyce Fee <[email protected]> * Update modules/get-started/pages/cluster-types/byoc/vpc-byo-aws.adoc Co-authored-by: Joyce Fee <[email protected]> * Update modules/get-started/pages/cluster-types/byoc/vpc-byo-aws.adoc Co-authored-by: Joyce Fee <[email protected]> * Update modules/get-started/pages/cluster-types/byoc/vpc-byo-aws.adoc Co-authored-by: Joyce Fee <[email protected]> * incorporate suggestions from doc review * BYOVPC for AWS (#47) * Insert newline before Prerequisites Header is not rendering due to missing newline. * Replace rpk user code with link to cloud-examples * Replace Configure your VPC section Modify this section to be the place where we refer the user to the cloud-examples repo for the terraform code. In the future this section can contain 2 tabs: one for the cloud-examples/terraform approach and one with aws cli commands. * Update the network/cluster create curl commands Some fields have been removed since this was written. Also fixed some syntax issues preventing the command vars from displaying properly. * Update the post-creation curl commands * Add information about rpk validation (#48) Adds the rpk validation output to the sample. Adds an explanation of what the validation will be checking. * doc edits * nest AWS, Azure, GCP folders under create BYOC also add aliases & update links to new locations * Update modules/get-started/pages/cluster-types/byoc/AWS/vpc-byo-aws.adoc Co-authored-by: Joyce Fee <[email protected]> * Update modules/get-started/pages/cluster-types/byoc/AWS/vpc-byo-aws.adoc Co-authored-by: Joyce Fee <[email protected]> * Update modules/get-started/pages/cluster-types/byoc/AWS/vpc-byo-aws.adoc Co-authored-by: Joyce Fee <[email protected]> * add glossterm * incorporate feedback from doc review * build with fixed glossterm * Updates to public API call 1. Remove redpanda_cloud_storage_manager_policy it is no longer used 2. Add "arn" prefix to connectors_node_group_instance_profile 3. Add permissions boundary field * incorporate feedback from Praseed + lowercase folders (in URL) * force folder rename * remove extra folders * re add vpc byo * add beta tag + minor text edits * edit What's New: BYOC VNet in LA --------- Co-authored-by: Joyce Fee <[email protected]> Co-authored-by: Sarah Haskins <[email protected]> Co-authored-by: Sarah Haskins <[email protected]> Co-authored-by: Paulo Borges <[email protected]>
- Loading branch information
5 people
authored
Dec 2, 2024
1 parent
f4cc281
commit c65341d
Showing
7 changed files
with
274 additions
and
8 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
261 changes: 261 additions & 0 deletions
261
modules/get-started/pages/cluster-types/byoc/aws/vpc-byo-aws.adoc
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,261 @@ | ||
| = Configure a Customer-Managed VPC on AWS | ||
| :description: Connect Redpanda Cloud to your existing VPC for additional security. | ||
| :page-aliases: get-started:cluster-types/byoc/vpc-byo-aws.adoc | ||
| :page-cloud: true | ||
| :page-beta: true | ||
|
|
||
| include::shared:partial$feature-flag.adoc[] | ||
|
|
||
| With a standard BYOC cluster, Redpanda manages the VPC lifecycle. For additional security, you can deploy the Redpanda glossterm:data plane[] into your existing shared VPC and manage the VPC lifecycle yourself. When you create a BYOC cluster, you specify your VPC and service account. The Redpanda Cloud agent doesn't create any new resources or alter any settings in your account. With a *customer-managed* VPC: | ||
|
|
||
| * You provide your own VPC in your AWS account. | ||
| * You maintain more control over your account, because Redpanda requires fewer permissions than standard BYOC clusters. | ||
| * You control your security resources and policies, including subnets, service accounts, IAM roles, firewall rules, and storage buckets. | ||
| == Prerequisites | ||
|
|
||
| * Familiarity with the xref:api:ROOT:cloud-api.adoc[Redpanda Cloud API]. For example, you should familiarize yourself with how to use the Cloud API to authenticate and create a cluster. | ||
| * Access to an AWS project in which you create your cluster. | ||
| * Minimum permissions in that AWS project. For the actions required by the user who will create the cluster with `rpk cloud byoc aws apply`, see https://github.com/redpanda-data/cloud-examples/blob/main/customer-managed/aws/terraform/iam_rpk_user.tf[`iam_rpk_user.tf`^]. | ||
|
|
||
| == Limitations | ||
|
|
||
| * You cannot move existing clusters to a customer-managed VPC. | ||
| * You cannot change to a different VPC after creating a cluster with a customer-managed VPC. | ||
|
|
||
| == Create managed resources | ||
|
|
||
| For example Terraform code of the expected provisioned resources, see the https://github.com/redpanda-data/cloud-examples/tree/main/customer-managed/aws[customer-managed VPC on AWS README^]. | ||
| The ARNs (Amazon Resource Names) output by this Terraform code are necessary in later steps. | ||
|
|
||
| == Create a network with the Cloud API | ||
|
|
||
| After xref:deploy:deployment-option/cloud/api/cloud-api-authentication.adoc[authenticating] to the Cloud API, issue a create network request using the Terraform outputs. | ||
|
|
||
| For information about the fields, see | ||
| https://redpanda-api-documentation.netlify.app/#post-/v1beta2/networks[Create Network]. | ||
|
|
||
| ``` | ||
| curl -X POST "https://api.redpanda.com/v1beta2/networks" \ | ||
| -H "accept: application/json"\ | ||
| -H "content-type: application/json" \ | ||
| -H "authorization: Bearer $YOUR_TOKEN" \ | ||
| -d @- << EOF | ||
| { | ||
| "name":"<your network resource name>", | ||
| "resource_group_id": "<resource group id of the network>", | ||
| "cloud_provider":"CLOUD_PROVIDER_AWS", | ||
| "region":"<region>", | ||
| "cluster_type":"TYPE_BYOC", | ||
| "customer_managed_resources": { | ||
| "aws": { | ||
| "management_bucket": { | ||
| "arn": "<management_bucket_arn from terraform outputs>" | ||
| }, | ||
| "dynamodb_table": { | ||
| "arn": "<dynamodb_table_arn from terraform outputs>" | ||
| }, | ||
| "private_subnets": { | ||
| "arns": [<private_subnet_ids from terraform outputs>] | ||
| }, | ||
| "vpc": { | ||
| "arn": "<vpc_arn from terraform outputs>" | ||
| } | ||
| } | ||
| } | ||
| } | ||
| EOF | ||
| ``` | ||
|
|
||
| The create network request returns a `resource_id`. For example: | ||
|
|
||
| [,yaml,lines=11] | ||
| ---- | ||
| { | ||
| "operation":{ | ||
| "id":"cpas8k6r4up5li18auh0", | ||
| "metadata":{ | ||
| "@type":"type.googleapis.com/redpanda.api.controlplane.v1beta2.CreateNetworkMetadata", | ||
| "network_id":"cpb338gekjj5i1cpj3t0" | ||
| }, | ||
| "state":"STATE_IN_PROGRESS", | ||
| "started_at":"2024-05-28T19:33:54.631Z", | ||
| "type":"TYPE_CREATE_NETWORK", | ||
| "resource_id":"cpb338gekjj5i1cpj3t0" | ||
| } | ||
| } | ||
| ---- | ||
|
|
||
| == Create a cluster with the Cloud API | ||
|
|
||
| To create a cluster, issue a create cluster request using the Terraform outputs. | ||
|
|
||
| For information about the fields, see | ||
| https://redpanda-api-documentation.netlify.app/#post-/v1beta2/clusters[Create Cluster]. | ||
|
|
||
| ``` | ||
| curl -X POST "https://api.redpanda.com/v1beta2/clusters" \ | ||
| -H "accept: application/json"\ | ||
| -H "content-type: application/json" \ | ||
| -H "authorization: Bearer $YOUR_TOKEN" \ | ||
| -d @- << EOF | ||
| { | ||
| "cloud_provider":"CLOUD_PROVIDER_AWS", | ||
| "connection_type":"CONNECTION_TYPE_PRIVATE", | ||
| "name":"<name of cluster>", | ||
| "resource_group_id":"<resource group id of the network>", | ||
| "network_id":"<resource_id of network from previous step>", | ||
| "region":"<region>", | ||
| "throughput_tier":"<throughput tier>", | ||
| "type":"TYPE_BYOC", | ||
| "zones":["<zone 1>", "<zone 2>", "<zone 3>"], | ||
| "redpanda_version": "<redpanda version>", | ||
| "customer_managed_resources": { | ||
| "aws": { | ||
| "agent_instance_profile": { | ||
| "arn": "<agent_instance_profile_arn from terraform outputs>" | ||
| }, | ||
| "connectors_node_group_instance_profile": { | ||
| "arn": "<connectors_node_group_instance_profile_arn from terraform outputs>" | ||
| }, | ||
| "redpanda_node_group_instance_profile": { | ||
| "arn": "<redpanda_node_group_instance_profile_arn from terraform outputs>" | ||
| }, | ||
| "utility_node_group_instance_profile": { | ||
| "arn": "<utility_node_group_instance_profile_arn from terraform outputs>" | ||
| }, | ||
| "connectors_security_group": { | ||
| "arn": "<connectors_security_group_arn from terraform outputs>" | ||
| }, | ||
| "node_security_group": { | ||
| "arn": "<node_security_group_arn from terraform outputs>" | ||
| }, | ||
| "utility_security_group": { | ||
| "arn": "<utility_security_group_arn from terraform outputs>" | ||
| }, | ||
| "redpanda_agent_security_group": { | ||
| "arn": "<redpanda_agent_security_group_arn from terraform outputs>" | ||
| }, | ||
| "redpanda_node_group_security_group": { | ||
| "arn": "<redpanda_node_group_security_group_arn from terraform outputs>" | ||
| }, | ||
| "cluster_security_group": { | ||
| "arn": "<cluster_security_group_arn from terraform outputs>" | ||
| }, | ||
| "k8s_cluster_role": { | ||
| "arn": "<k8s_cluster_role_arn from terraform outputs>" | ||
| }, | ||
| "cloud_storage_bucket": { | ||
| "arn": "<cloud_storage_bucket_arn from terraform outputs>" | ||
| }, | ||
| "permissions_boundary_policy": { | ||
| "arn": "<permissions_boundary_policy_arn from terraform outputs>" | ||
| } | ||
| } | ||
| }, | ||
| # <The following aws_private_link section is optional. For more information, see https://docs.redpanda.com/current/deploy/deployment-option/cloud/aws-privatelink/.> | ||
| "aws_private_link": { | ||
| "enabled": true, | ||
| "allowed_principals": [<allowed principals>], | ||
| "connect_console": <true|false> | ||
| } | ||
| } | ||
| EOF | ||
| ``` | ||
|
|
||
| The create cluster request returns a `resource_id`, which is required in the next step. For example: | ||
|
|
||
| ```bash | ||
| { | ||
| "operation":{ | ||
| "id":"cpas8k6r4up5li18auhg", | ||
| "metadata":{ | ||
| "@type":"type.googleapis.com/redpanda.api.controlplane.v1beta2.CreateClusterMetadata", | ||
| "cluster_id":"cpb33c8ekjj5i1cpj3v0" | ||
| }, | ||
| "state":"STATE_IN_PROGRESS", | ||
| "started_at":"2024-05-28T19:34:09.501Z", | ||
| "type":"TYPE_CREATE_CLUSTER", | ||
| "resource_id":"cpb33c8ekjj5i1cpj3v0" | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| == Create cluster resources | ||
|
|
||
| To create the initial cluster resources, run `rpk cloud byoc aws apply`. This creates an autoscaling group, an agent VM, and the following resources: | ||
|
|
||
| * S3 objects | ||
| * Launch template | ||
| * Autoscaling group | ||
|
|
||
| NOTE: You must have the `iam_rpk_user.tf` permissions described in the prerequisites. | ||
|
|
||
| ```bash | ||
| rpk cloud login \ | ||
| --save \ | ||
| --client-id='<client-id>’ \ | ||
| --client-secret='<client-secret>' \ | ||
| --no-profile | ||
|
|
||
| rpk cloud byoc aws apply \ | ||
| --redpanda-id='<resource_id of cluster from previous step>' | ||
| ``` | ||
|
|
||
| Output: | ||
|
|
||
| ```bash | ||
| Checking RPK User... PASSED | ||
| Checking IAM Instance Profiles... PASSED | ||
| Checking Storage... PASSED | ||
| Checking Network... PASSED | ||
| Reconciling agent infrastructure... | ||
| Running apply {"provisioner": "redpanda-bootstrap"} | ||
| Finished apply {"provisioner": "redpanda-bootstrap"} | ||
| Running apply {"provisioner": "redpanda-network"} | ||
| Finished apply {"provisioner": "redpanda-network"} | ||
| Running apply {"provisioner": "redpanda-agent"} | ||
| Finished apply {"provisioner": "redpanda-agent"} | ||
| The Redpanda cluster is deploying. This can take up to 45 minutes. View status at https://cloud.redpanda.com/clusters/<resource_id of cluster from previous step>/overview. | ||
| ``` | ||
|
|
||
| The agent VM now is running and handles the remaining provisioning steps. This can take up to 45 minutes. When provisioning completes, the cluster status updates to `Running`. If the cluster remains in `Creating` status after 45 minutes, contact https://support.redpanda.com/hc/en-us/requests/new[Redpanda support^]. | ||
|
|
||
| === Validatation checks | ||
|
|
||
| The `rpk cloud byoc aws apply` command performs validation checks before proceeding with provisioning: | ||
|
|
||
| * RPK user: Checks if the user running the command has sufficient privileges to provision the agent. Any | ||
| missing permissions are displayed in the output. | ||
|
|
||
| * IAM instance profile: Checks that `connectors_node_group_instance_profile`, `redpanda_node_group_instance_profile`, | ||
| `utility_node_group_instance_profile`, and `k8s_cluster_role` have the minimum required permissions. Any missing permissions are displayed in the output. | ||
|
|
||
| * Storage: Checks that the `management_bucket` exists and is versioned, checks that the `cloud_storage_bucket` exists and is not versioned, and checks that the `dynamodb_table` exists. | ||
|
|
||
| * Network: Checks that the VPC exists, checks that the subnets exist and have the expected tags, and checks that the security groups exist and have the desired ingress and egress rules. | ||
|
|
||
| If you think validation errors are erroneous, you can rerun the command with the `--no-validate` tag | ||
| to skip validation. | ||
|
|
||
| == Check cluster status | ||
|
|
||
| You can check the cluster status with the Cloud API or the Redpanda Cloud UI. | ||
|
|
||
| Example using the returned `operation_id`: | ||
|
|
||
| ```bash | ||
| curl -X GET "https://api.redpanda.com/v1beta2/operations/<operation_id of operation from previous step>" \ | ||
| -H "accept: application/json"\ | ||
| -H "content-type: application/json" \ | ||
| -H "authorization: Bearer $YOUR_TOKEN" | ||
| ``` | ||
|
|
||
| Example retrieving cluster: | ||
|
|
||
| ```bash | ||
| curl -X GET "https://api.redpanda.com/v1beta2/clusters/<resource_id of cluster from previous step>" \ | ||
| -H "accept: application/json"\ | ||
| -H "content-type: application/json" \ | ||
| -H "authorization: Bearer $YOUR_TOKEN" | ||
| ``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters