Confluent - Cluster Module

About this repository

Terraform module which creates Confluent Kafka Cluster with Topic

About Confluent

Confluent Platform is a full-scale data streaming platform that enables you to easily access, store, and manage data as continuous, real-time streams. Built by the original creators of Apache Kafka®, Confluent expands the benefits of Kafka with enterprise-grade features while removing the burden of Kafka management or monitoring. Today, over 80% of the Fortune 100 are powered by data streaming technology – and the majority of those leverage Confluent.

Usage

module "confluent_cluster" {
  source                   = "github.com/jveraduran/terraform-confluent-module//module/cluster?ref=v1.0.0"
  display_name             = "demo"
  availability             = "SINGLE_ZONE"
  cloud                    = "AWS"
  region                   = "us-east-2"
  create_basic_cluster     = var.create_basic_cluster
  create_standard_cluster  = var.create_standard_cluster
  create_dedicated_cluster = var.create_dedicated_cluster
  create_api_key           = true
}

module "confluent_topics" {
  depends_on = [
    module.confluent_cluster
  ]
  source           = "github.com/jveraduran/terraform-confluent-module//module/topics?ref=v1.0.0"
  topic_name       = "demo"
  partitions_count = 4
  cluster_id       = var.create_basic_cluster == true ? module.confluent_cluster.basic_cluster_id[0] : (var.create_standard_cluster == true ? module.confluent_cluster.standard_cluster_id[0] : module.confluent_cluster.dedicated_cluster_id[0])
  environment      = module.confluent_cluster.confluent_environment_id
  api_key          = module.confluent_cluster.api_key[0]
  api_secret       = module.confluent_cluster.api_secret[0]
}

Examples

Basic Cluster AWS | Basic Cluster GCP | Basic Cluster AZURE Basic clusters are designed for development use-cases. Basic clusters support the following:

• 99.5% uptime SLA
• Up to 100 MB/s of throughput and 5 TB of storage.
• You only pay for the ingress, egress, storage, and partitions. There is no base cluster price.
• Can be upgraded to a single-zone Standard cluster at any time using the Confluent Cloud Console.

Standard Cluster AWS | Standard Cluster GCP | Standard Cluster AZURE

• Uptime SLA: 99.95% for Single-Zone, 99.99% for Multi-Zone
• Up to 100 MB/s of throughput and unlimited storage.
• Multi-zone high availability (optional). A multi-zone cluster is spread across three availability zones for better resiliency.
• Charged an hourly base price in addition to the ingress, egress, storage, and partitions.

Dedicated Cluster AWS | Dedicated Cluster GCP | Dedicated Cluster AZURE

• Single-tenant deployments with a 99.95% uptime SLA for Single-Zone, and 99.99% for Multi-Zone
• Private networking options including VPC peering, AWS Transit Gateway, AWS PrivateLink, and Azure PrivateLink.
• Self-managed keys when AWS or Google Cloud is the cloud service provider.
• Multi-zone high availability (optional). A multi-zone cluster is spread across three availability zones for better resiliency.
• Can be scaled to achieve gigabytes per second of ingress.
• Simple scaling in terms of CKUs.
• Cluster expansion, and Cluster shrinking.

Contributing

I'm grateful to the community for contributing bugfixes and improvements! Please see below to learn how you can take part.

• Code of Conduct
• Contributing Guide

Requirements

Name	Version
confluent	1.4.0

Providers

Name	Version
confluent	1.4.0

Modules

Name	Source	Version
confluent_cluster	github.com/jveraduran/terraform-confluent-module//module/cluster	v1.0.0
confluent_topics	github.com/jveraduran/terraform-confluent-module//module/topics	v1.0.0

Resources

Name	Type
confluent_api_key.main	resource
confluent_environment.main	resource
confluent_kafka_cluster.basic	resource
confluent_kafka_cluster.dedicated	resource
confluent_kafka_cluster.standard	resource
confluent_role_binding.main	resource
confluent_service_account.main	resource
confluent_kafka_topic.main	resource
confluent_kafka_cluster.main	data source

Inputs

Name	Description	Type	Default	Required
availability	n/a	string	null	no
cloud	n/a	string	null	no
cloud_api_key	n/a	string	n/a	yes
cloud_api_secret	n/a	string	n/a	yes
create_basic_cluster	n/a	bool	false	no
create_dedicated_cluster	n/a	bool	false	no
create_standard_cluster	n/a	bool	true	no
display_name	n/a	string	null	no
region	n/a	string	null	no

Outputs

Name	Description
api_key	(Required String) The ID of the API Key, for example, EGWX3S4BVNQIRBMJ
api_secret	(Required String, Sensitive) The secret of the API Key.
basic_cluster_api_kind	(Required String) A kind of the Kafka cluster, for example, Cluster.
basic_cluster_api_version	(Required String) An API Version of the schema version of the Kafka cluster, for example, cmk/v2.
basic_cluster_bootstrap_endpoint	(Required String) The bootstrap endpoint used by Kafka clients to connect to the Kafka cluster. (e.g., SASL_SSL://pkc-00000.us-central1.gcp.confluent.cloud:9092).
basic_cluster_id	(Required String) The ID of the Kafka cluster (e.g., lkc-abc123).
basic_cluster_rbac_crn	(Required String) The Confluent Resource Name of the Kafka cluster, for example, crn://confluent.cloud/organization=1111aaaa-11aa-11aa-11aa-111111aaaaaa/environment=env-abc123/cloud-cluster=lkc-abc123
basic_cluster_rest_endpoint	(Required String) The REST endpoint of the Kafka cluster (e.g., https://pkc-00000.us-central1.gcp.confluent.cloud:443).
confluent_environment_display_name	The Display Name of the Environment
confluent_environment_id	The ID of the Environment, for example, env-abc123
confluent_kafka_topic_id	(Required String) The ID of the Kafka topic, in the format /, for example, lkc-abc123/orders-1
confluent_role_binding_id	(Required String) The ID of the Role Binding (e.g., rb-f3a90de).
confluent_service_account_api_version	(Required String) An API Version of the schema version of the Service Account, for example, iam/v2.
confluent_service_account_id	(Required String) The ID of the Service Account (e.g., sa-abc123).
confluent_service_account_kind	(Required String) A kind of the Service Account, for example, ServiceAccount
dedicated_cluster_api_kind	(Required String) A kind of the Kafka cluster, for example, Cluster.
dedicated_cluster_api_version	(Required String) An API Version of the schema version of the Kafka cluster, for example, cmk/v2.
dedicated_cluster_bootstrap_endpoint	(Required String) The bootstrap endpoint used by Kafka clients to connect to the Kafka cluster. (e.g., SASL_SSL://pkc-00000.us-central1.gcp.confluent.cloud:9092).
dedicated_cluster_id	(Required String) The ID of the Kafka cluster (e.g., lkc-abc123).
dedicated_cluster_rbac_crn	(Required String) The Confluent Resource Name of the Kafka cluster, for example, crn://confluent.cloud/organization=1111aaaa-11aa-11aa-11aa-111111aaaaaa/environment=env-abc123/cloud-cluster=lkc-abc123
dedicated_cluster_rest_endpoint	(Required String) The REST endpoint of the Kafka cluster (e.g., https://pkc-00000.us-central1.gcp.confluent.cloud:443).
standard_cluster_api_kind	(Required String) A kind of the Kafka cluster, for example, Cluster.
standard_cluster_api_version	(Required String) An API Version of the schema version of the Kafka cluster, for example, cmk/v2.
standard_cluster_bootstrap_endpoint	(Required String) The bootstrap endpoint used by Kafka clients to connect to the Kafka cluster. (e.g., SASL_SSL://pkc-00000.us-central1.gcp.confluent.cloud:9092).
standard_cluster_id	(Required String) The ID of the Kafka cluster (e.g., lkc-abc123).
standard_cluster_rbac_crn	(Required String) The Confluent Resource Name of the Kafka cluster, for example, crn://confluent.cloud/organization=1111aaaa-11aa-11aa-11aa-111111aaaaaa/environment=env-abc123/cloud-cluster=lkc-abc123
standard_cluster_rest_endpoint	(Required String) The REST endpoint of the Kafka cluster (e.g., https://pkc-00000.us-central1.gcp.confluent.cloud:443).