diff --git a/README.adoc b/README.adoc index e95326d1..bb0591b1 100644 --- a/README.adoc +++ b/README.adoc @@ -14,7 +14,7 @@ image:https://api.netlify.com/api/v1/badges/5b89dd6f-1847-419c-b3be-a1650ce8992f
++++ -This repository hosts the documentation content for Redpanda Platform and Redpanda Cloud. +This repository hosts the documentation content for Redpanda. == Contribute @@ -97,7 +97,6 @@ Versioned content is stored in branches that track the version of Redpanda Platf The production {url-playbook}[docs site playbook] instructs Antora to automatically aggregate content in the following branches: - `main`: Content for the latest version of Redpanda. -- `v/*`: Content for previous versions of Redpanda. - `shared`: Content that is shared across all versions (asciidoc attributes and terms) - `api`: API docs as well as the source OpenAPI spec files. - `site-search`: The global site search page. diff --git a/modules/ROOT/nav.adoc b/modules/ROOT/nav.adoc index 2415ec14..23c8cb85 100644 --- a/modules/ROOT/nav.adoc +++ b/modules/ROOT/nav.adoc @@ -57,19 +57,34 @@ ** xref:ROOT:develop:config-topics.adoc[] ** xref:ROOT:develop:http-proxy.adoc[] ** xref:ROOT:develop:transactions.adoc[] +** xref:develop:managed-connectors/index.adoc[Managed Connectors] +*** xref:develop:managed-connectors/converters-and-serialization.adoc[Converters and serialization] +*** xref:develop:managed-connectors/monitor-connectors.adoc[Monitor Connectors] +*** xref:develop:managed-connectors/transforms.adoc[Single Message Transforms] +*** xref:develop:managed-connectors/sizing-connectors.adoc[Sizing Connectors] +*** xref:develop:managed-connectors/create-s3-sink-connector.adoc[AWS S3 Sink Connector] +*** xref:develop:managed-connectors/create-gcp-bigquery-connector.adoc[Google BigQuery Sink Connector] +*** xref:develop:managed-connectors/create-gcs-connector.adoc[GCS Sink Connector] +*** xref:develop:managed-connectors/create-http-source-connector.adoc[HTTP Source Connector] +*** xref:develop:managed-connectors/create-iceberg-sink-connector.adoc[Iceberg Sink Connector] +*** xref:develop:managed-connectors/create-jdbc-sink-connector.adoc[JDBC Sink Connector] +*** xref:develop:managed-connectors/create-jdbc-source-connector.adoc[JDBC Source Connector] +*** xref:develop:managed-connectors/create-mmaker-source-connector.adoc[MirrorMaker2 Source Connector] +*** xref:develop:managed-connectors/create-mmaker-checkpoint-connector.adoc[MirrorMaker2 Checkpoint Connector] +*** xref:develop:managed-connectors/create-mmaker-heartbeat-connector.adoc[MirrorMaker2 Heartbeat Connector] +*** xref:develop:managed-connectors/create-mongodb-sink-connector.adoc[MongoDB Sink Connector] +*** xref:develop:managed-connectors/create-mongodb-source-connector.adoc[MongoDB Source Connector] +*** xref:develop:managed-connectors/create-mysql-source-connector.adoc[MySQL (Debezium) Source Connector] +*** xref:develop:managed-connectors/create-postgresql-connector.adoc[PostgreSQL (Debezium) Source Connector] +*** xref:develop:managed-connectors/create-snowflake-connector.adoc[Snowflake Sink Connector] * xref:manage:index.adoc[Manage] ** xref:manage:monitor-cloud.adoc[] ** xref:ROOT:get-started:rpk/index.adoc[Redpanda CLI] -** xref:ROOT:get-started:intro-to-rpk.adoc[] +*** xref:ROOT:get-started:intro-to-rpk.adoc[] *** xref:ROOT:get-started:broker-admin.adoc[] *** xref:ROOT:get-started:rpk-install.adoc[] *** xref:ROOT:get-started:config-rpk-profile.adoc[] -** xref:manage:api/index.adoc[Cloud API] -*** xref:manage:api/cloud-api-overview.adoc[Cloud API Overview] -*** xref:manage:api/cloud-api-authentication.adoc[Cloud API Authentication] -*** xref:manage:api/cloud-controlplane-api.adoc[Use the Control Plane API] -*** xref:manage:api/cloud-dataplane-api.adoc[Use the Data Plane APIs] ** xref:manage:schema-reg/index.adoc[Schema Registry] *** xref:ROOT:manage:schema-reg/schema-reg-overview.adoc[] *** xref:ROOT:manage:schema-reg/schema-reg-ui.adoc[Use Schema Registry in the Cloud UI] @@ -78,26 +93,14 @@ *** xref:ROOT:reference:console/record-deserialization.adoc[Deserialization] *** xref:ROOT:reference:console/programmable-push-filters.adoc[Programmable Push Filters] *** xref:ROOT:manage:console/edit-topic-configuration.adoc[Edit Topic Configuration] -** xref:manage:managed-connectors/index.adoc[Managed Connectors] -*** xref:manage:managed-connectors/converters-and-serialization.adoc[Converters and serialization] -*** xref:manage:managed-connectors/monitor-connectors.adoc[Monitor Connectors] -*** xref:manage:managed-connectors/transforms.adoc[Single Message Transforms] -*** xref:manage:managed-connectors/sizing-connectors.adoc[Sizing Connectors] -*** xref:manage:managed-connectors/create-s3-sink-connector.adoc[AWS S3 Sink Connector] -*** xref:manage:managed-connectors/create-gcp-bigquery-connector.adoc[Google BigQuery Sink Connector] -*** xref:manage:managed-connectors/create-gcs-connector.adoc[GCS Sink Connector] -*** xref:manage:managed-connectors/create-http-source-connector.adoc[HTTP Source Connector] -*** xref:manage:managed-connectors/create-iceberg-sink-connector.adoc[Iceberg Sink Connector] -*** xref:manage:managed-connectors/create-jdbc-sink-connector.adoc[JDBC Sink Connector] -*** xref:manage:managed-connectors/create-jdbc-source-connector.adoc[JDBC Source Connector] -*** xref:manage:managed-connectors/create-mmaker-source-connector.adoc[MirrorMaker2 Source Connector] -*** xref:manage:managed-connectors/create-mmaker-checkpoint-connector.adoc[MirrorMaker2 Checkpoint Connector] -*** xref:manage:managed-connectors/create-mmaker-heartbeat-connector.adoc[MirrorMaker2 Heartbeat Connector] -*** xref:manage:managed-connectors/create-mongodb-sink-connector.adoc[MongoDB Sink Connector] -*** xref:manage:managed-connectors/create-mongodb-source-connector.adoc[MongoDB Source Connector] -*** xref:manage:managed-connectors/create-mysql-source-connector.adoc[MySQL (Debezium) Source Connector] -*** xref:manage:managed-connectors/create-postgresql-connector.adoc[PostgreSQL (Debezium) Source Connector] -*** xref:manage:managed-connectors/create-snowflake-connector.adoc[Snowflake Sink Connector] +** xref:manage:api/index.adoc[Cloud API] +*** xref:manage:api/cloud-api-quickstart.adoc[Cloud API Quickstart] +*** xref:manage:api/cloud-api-overview.adoc[Cloud API Overview] +*** xref:manage:api/cloud-api-authentication.adoc[Cloud API Authentication] +*** xref:manage:api/cloud-controlplane-api.adoc[Use the Control Plane API] +*** xref:manage:api/cloud-dataplane-api.adoc[Use the Data Plane APIs] +*** xref:manage:api/cloud-api-errors.adoc[Errors and Status Codes] +** xref:manage:remote-read-replicas.adoc[] * xref:billing:index.adoc[Billing] ** xref:billing:billing.adoc[] diff --git a/modules/manage/pages/managed-connectors/converters-and-serialization.adoc b/modules/develop/pages/managed-connectors/converters-and-serialization.adoc similarity index 100% rename from modules/manage/pages/managed-connectors/converters-and-serialization.adoc rename to modules/develop/pages/managed-connectors/converters-and-serialization.adoc diff --git a/modules/manage/pages/managed-connectors/create-gcp-bigquery-connector.adoc b/modules/develop/pages/managed-connectors/create-gcp-bigquery-connector.adoc similarity index 100% rename from modules/manage/pages/managed-connectors/create-gcp-bigquery-connector.adoc rename to modules/develop/pages/managed-connectors/create-gcp-bigquery-connector.adoc diff --git a/modules/manage/pages/managed-connectors/create-gcs-connector.adoc b/modules/develop/pages/managed-connectors/create-gcs-connector.adoc similarity index 100% rename from modules/manage/pages/managed-connectors/create-gcs-connector.adoc rename to modules/develop/pages/managed-connectors/create-gcs-connector.adoc diff --git a/modules/manage/pages/managed-connectors/create-http-source-connector.adoc b/modules/develop/pages/managed-connectors/create-http-source-connector.adoc similarity index 100% rename from modules/manage/pages/managed-connectors/create-http-source-connector.adoc rename to modules/develop/pages/managed-connectors/create-http-source-connector.adoc diff --git a/modules/manage/pages/managed-connectors/create-iceberg-sink-connector.adoc b/modules/develop/pages/managed-connectors/create-iceberg-sink-connector.adoc similarity index 100% rename from modules/manage/pages/managed-connectors/create-iceberg-sink-connector.adoc rename to modules/develop/pages/managed-connectors/create-iceberg-sink-connector.adoc diff --git a/modules/manage/pages/managed-connectors/create-jdbc-sink-connector.adoc b/modules/develop/pages/managed-connectors/create-jdbc-sink-connector.adoc similarity index 100% rename from modules/manage/pages/managed-connectors/create-jdbc-sink-connector.adoc rename to modules/develop/pages/managed-connectors/create-jdbc-sink-connector.adoc diff --git a/modules/manage/pages/managed-connectors/create-jdbc-source-connector.adoc b/modules/develop/pages/managed-connectors/create-jdbc-source-connector.adoc similarity index 100% rename from modules/manage/pages/managed-connectors/create-jdbc-source-connector.adoc rename to modules/develop/pages/managed-connectors/create-jdbc-source-connector.adoc diff --git a/modules/manage/pages/managed-connectors/create-mmaker-checkpoint-connector.adoc b/modules/develop/pages/managed-connectors/create-mmaker-checkpoint-connector.adoc similarity index 100% rename from modules/manage/pages/managed-connectors/create-mmaker-checkpoint-connector.adoc rename to modules/develop/pages/managed-connectors/create-mmaker-checkpoint-connector.adoc diff --git a/modules/manage/pages/managed-connectors/create-mmaker-heartbeat-connector.adoc b/modules/develop/pages/managed-connectors/create-mmaker-heartbeat-connector.adoc similarity index 100% rename from modules/manage/pages/managed-connectors/create-mmaker-heartbeat-connector.adoc rename to modules/develop/pages/managed-connectors/create-mmaker-heartbeat-connector.adoc diff --git a/modules/manage/pages/managed-connectors/create-mmaker-source-connector.adoc b/modules/develop/pages/managed-connectors/create-mmaker-source-connector.adoc similarity index 100% rename from modules/manage/pages/managed-connectors/create-mmaker-source-connector.adoc rename to modules/develop/pages/managed-connectors/create-mmaker-source-connector.adoc diff --git a/modules/manage/pages/managed-connectors/create-mongodb-sink-connector.adoc b/modules/develop/pages/managed-connectors/create-mongodb-sink-connector.adoc similarity index 100% rename from modules/manage/pages/managed-connectors/create-mongodb-sink-connector.adoc rename to modules/develop/pages/managed-connectors/create-mongodb-sink-connector.adoc diff --git a/modules/manage/pages/managed-connectors/create-mongodb-source-connector.adoc b/modules/develop/pages/managed-connectors/create-mongodb-source-connector.adoc similarity index 100% rename from modules/manage/pages/managed-connectors/create-mongodb-source-connector.adoc rename to modules/develop/pages/managed-connectors/create-mongodb-source-connector.adoc diff --git a/modules/manage/pages/managed-connectors/create-mysql-source-connector.adoc b/modules/develop/pages/managed-connectors/create-mysql-source-connector.adoc similarity index 100% rename from modules/manage/pages/managed-connectors/create-mysql-source-connector.adoc rename to modules/develop/pages/managed-connectors/create-mysql-source-connector.adoc diff --git a/modules/manage/pages/managed-connectors/create-postgresql-connector.adoc b/modules/develop/pages/managed-connectors/create-postgresql-connector.adoc similarity index 100% rename from modules/manage/pages/managed-connectors/create-postgresql-connector.adoc rename to modules/develop/pages/managed-connectors/create-postgresql-connector.adoc diff --git a/modules/manage/pages/managed-connectors/create-s3-sink-connector.adoc b/modules/develop/pages/managed-connectors/create-s3-sink-connector.adoc similarity index 100% rename from modules/manage/pages/managed-connectors/create-s3-sink-connector.adoc rename to modules/develop/pages/managed-connectors/create-s3-sink-connector.adoc diff --git a/modules/manage/pages/managed-connectors/create-snowflake-connector.adoc b/modules/develop/pages/managed-connectors/create-snowflake-connector.adoc similarity index 100% rename from modules/manage/pages/managed-connectors/create-snowflake-connector.adoc rename to modules/develop/pages/managed-connectors/create-snowflake-connector.adoc diff --git a/modules/manage/pages/managed-connectors/index.adoc b/modules/develop/pages/managed-connectors/index.adoc similarity index 100% rename from modules/manage/pages/managed-connectors/index.adoc rename to modules/develop/pages/managed-connectors/index.adoc diff --git a/modules/manage/pages/managed-connectors/monitor-connectors.adoc b/modules/develop/pages/managed-connectors/monitor-connectors.adoc similarity index 100% rename from modules/manage/pages/managed-connectors/monitor-connectors.adoc rename to modules/develop/pages/managed-connectors/monitor-connectors.adoc diff --git a/modules/manage/pages/managed-connectors/sizing-connectors.adoc b/modules/develop/pages/managed-connectors/sizing-connectors.adoc similarity index 100% rename from modules/manage/pages/managed-connectors/sizing-connectors.adoc rename to modules/develop/pages/managed-connectors/sizing-connectors.adoc diff --git a/modules/manage/pages/managed-connectors/transforms.adoc b/modules/develop/pages/managed-connectors/transforms.adoc similarity index 100% rename from modules/manage/pages/managed-connectors/transforms.adoc rename to modules/develop/pages/managed-connectors/transforms.adoc diff --git a/modules/get-started/pages/whats-new-cloud.adoc b/modules/get-started/pages/whats-new-cloud.adoc index 1b8f2c93..b5baf3b1 100644 --- a/modules/get-started/pages/whats-new-cloud.adoc +++ b/modules/get-started/pages/whats-new-cloud.adoc @@ -5,6 +5,16 @@ This page lists new features added in Redpanda Cloud. +== June 2024 + +=== Remote read replica topics on BYOC: beta + +You can now create xref:manage:remote-read-replicas.adoc[remote read replica topics] on a BYOC cluster with the Cloud API. A remote read replica topic is a read-only topic that mirrors a topic on a different cluster. It can serve any consumer, without increasing the load on the source cluster. + +=== Higher connection limits in usage tiers + +Redpanda has increased the number of client connections in all xref:reference:tiers/byoc-tiers.adoc[tiers]. For example, tier 1 now supports up to 9,000 maximum connections, and tier 9 supports up to 450,000 maximum connections. Connections are regulated per broker for best performance. + == May 2024 === Cloud API: beta diff --git a/modules/manage/pages/api/index.adoc b/modules/manage/pages/api/index.adoc index 6d80db70..f121b5ca 100644 --- a/modules/manage/pages/api/index.adoc +++ b/modules/manage/pages/api/index.adoc @@ -2,4 +2,5 @@ :description: Use REST APIs to manage Redpanda Cloud resources. :page-layout: index :page-cloud: true +:page-beta: true //:page-categories: \ No newline at end of file diff --git a/modules/manage/pages/remote-read-replicas.adoc b/modules/manage/pages/remote-read-replicas.adoc new file mode 100644 index 00000000..71724c1b --- /dev/null +++ b/modules/manage/pages/remote-read-replicas.adoc @@ -0,0 +1,59 @@ += Remote Read Replicas +:description: Learn how to create a remote read replica topic, which is a read-only topic that mirrors a topic on a different cluster. +:page-cloud: true +:page-beta: true + +A remote read replica topic is a read-only topic that mirrors a topic on a different cluster. You can create a separate remote cluster just for consumers of this topic and populate its topics from object storage. A read-only topic on a remote cluster can serve any consumer, without increasing the load on the source cluster. Because these read-only topics access data directly from object storage, there's no impact to the performance of the cluster. + +Redpanda Cloud supports remote read replicas with ephemeral BYOC clusters (not with a customer-managed VPC). Ephemeral clusters are temporary clusters created to handle specific, transient workloads or tasks. After processing, the cluster is terminated to free up resources and reduce costs. + +== Prerequisites + +* A BYOC source cluster in Ready state. +* A BYOC reader cluster in Ready state. This separate reader cluster must exist in the same Redpanda organization and the same cloud provider account and region as the source cluster. + +== Configure remote read replicas + +You can add or remove reader clusters to a source cluster in Redpanda Cloud with the xref:api:ROOT:cloud-api.adoc[Cloud API]. For information on accessing the Cloud API, see xref:manage:api/cloud-api-authentication.adoc[]. + +. To update your source cluster to add one or more reader cluster IDs, make a xref:api:ROOT:cloud-api.adoc#patch-/v1beta2/clusters/-cluster.id-[`PATCH /v1beta2/clusters/\{cluster.id}`] request. The full list of clusters is expected on every call. If an ID is removed from the list, it will be removed as a reader cluster. ++ +```bash +export SOURCE_CLUSTER_ID=....... +export READER_CLUSTER_ID=....... + +curl -X PATCH $API_HOST/v1beta2/clusters/$SOURCE_CLUSTER_ID \ +-H "Content-Type: application/json" \ +-H "Authorization: Bearer $API_TOKEN" \ +-d @- << EOF +{ + "read_replica_cluster_ids": ["$READER_CLUSTER_ID"] +} +EOF +``` + +. To create a remote read replica topic, run: ++ +```bash +rpk topic create my-topic -c redpanda.remote.readreplica=redpanda-cloud-storage-${SOURCE_CLUSTER_ID} +``` + +. Optional: To see the list of reader clusters on a given source cluster, make a xref:api:ROOT:cloud-api.adoc#get-/v1beta2/clusters/-id-[`GET /v1beta2/clusters/\{id}`] request: ++ +```bash +export SOURCE_CLUSTER_ID=....... + +curl -X GET $API_HOST/v1beta2/clusters/$SOURCE_CLUSTER_ID \ +-H "Content-Type: application/json" \ +-H "Authorization: Bearer $API_TOKEN" +``` + +. Optional: For remote read replicas reading from a live topic (that is, a topic that's being actively written to by a source cluster), it may be advantageous to control how often segments are flushed to object storage. By default, this is set to 60 minutes. To tune `cloud_storage_segment_max_upload_interval_sec` on the source cluster, contact https://support.redpanda.com/hc/en-us/requests/new[Redpanda support^]. (For cold topics, where segments are closed and older than 60 minutes, this configuration is unnecessary: the data is already uploaded to object storage.) + +[NOTE] +==== +A source cluster cannot be deleted if it has remote read replica topics. When you delete a reader cluster, that cluster's ID is removed from any existing source cluster `read_replica_cluster_ids` lists. + +==== + + diff --git a/modules/reference/pages/rpk/index.adoc b/modules/reference/pages/rpk/index.adoc index dc5057fb..cca40563 100644 --- a/modules/reference/pages/rpk/index.adoc +++ b/modules/reference/pages/rpk/index.adoc @@ -1,5 +1,9 @@ = rpk Commands -:description: This page contains an alphabetized list of `rpk` commands. Each command includes a table of flags and their descriptions. +:description: pass:q[Index page of `rpk` commands for Redpanda Cloud in alphabetical order.] -You can also get descriptions for each flag by running `rpk