Zookeeper Connection Plaintext (Public Amazon MSK) not available?

[arinhouck]

I am able to run the following within my Codebuild instance within Amazon VPC using Kafka CLI to change Zookeeper ACL and topic.

Create ACL

./kafka-acls.sh --authorizer-properties zookeeper.connect=some-domain.us-east-1.amazonaws.com:2181 --add --allow-principal "User:scramuser" --operation All --group=* --topic *

Create Topic

./kafka-topics.sh --create --zookeeper some-domain.us-east-1.amazonaws.com:2181 --replication-factor 2 --partitions 2 --topic agent_index

However, when configuring via terraform it always returns this error:

kafka: client has run out of available brokers to talk to: EOF

using

terraform {
  required_providers {
    kafka = {
      source = "Mongey/kafka"
    }
  }
}

provider "kafka" {
  bootstrap_servers = split(",", var.servers)
  tls_enabled = false
}

resource "kafka_acl" "main" {
  resource_name       = "*"
  resource_type       = "Topic"
  acl_principal       = "User:${var.scram_username}"
  acl_host            = "*"
  acl_operation       = "All"
  acl_permission_type = "Allow"
}

resource "kafka_topic" "agent_index" {
  name               = "agent_index"
  replication_factor = 2
  partitions         = 2
}

where var.servers = "some-domain.us-east-1.amazonaws.com:2181" and var.scram_user = "scramuser".

So I have special flags assigned for zookeeper config as you can see in the commands above. Does this library only allow bootstrap servers? I don't see any documentation for using zookeeper config.

Following practices from following AWS docs:
https://docs.aws.amazon.com/msk/latest/developerguide/msk-acls.html
https://docs.aws.amazon.com/msk/latest/developerguide/mkc-create-topic.html

My goal is to streamline this process using terraform instead of managing sh scripts. However unsure if that is achievable. I actually still don't understand what the difference is for bootstrap server vs zookeeper server. I guess zookeeper is like a middleware. I doesn't seem Amazon provides any other option other than zookeeper for modifying topics and ACL in documentation.

EDIT: I see zookeeper isn't supported according to discussions in past issues of this repo.

I have tried SASL SCRAM-512 which works on public endpoint for consumer and producer on making connection. However for the private endpoint when running terraform apply through this library with SCRAM configured it doesn't seem to work even with all traffic allowed through VPC and in a VPC enabled environment (same instance that private zookeeper dns worked for).

I have verified in the variables using codebuild-breakpoint that the variables are correctly assigned for username and password.

TF_VAR_servers
TF_VAR_scram_username
TF_VAR_scram_password

provider "kafka" {
bootstrap_servers = split(",", var.servers)
tls_enabled = false
sasl_username = var.scram_username
sasl_password = var.scram_password
sasl_mechanism = "scram-sha512"
}

using urls provided with port 9096.

[arinhouck]

From what I can gather, zookeeper is the only option with Amazon MSK for managing ACLs. So it doesn't seem this is supporting Amazon MSK at all if I am not mistaken. Has anyone got this working with MSK?

[hugolesta]

@arinhouck, I was able to create ACL over MKS using plaintext

According to your code I can suggest turning on skip_tls_verify attribute to true, and trying again.

I'd suggest setting up the provider in the following way.

provider "kafka" {
  bootstrap_servers = split(",", var.servers)
  tls_enabled       = false
  skip_tls_verify   = true
}

[qq304635576]

@hugolesta if create ACL over MSK using SSL, It's failed. so do you have solution for that? As we know, considering security requirement, in common, using SSL is required in Production Environment.thanks

[qq304635576]

@arinhouck have any update? I also encounter the same issue with you.

[arinhouck]

@arinhouck, I was able to create ACL over MKS using plaintext

According to your code I can suggest turning on skip_tls_verify attribute to true, and trying again.

I'd suggest setting up the provider in the following way.

provider "kafka" {
  bootstrap_servers = split(",", var.servers)
  tls_enabled       = false
  skip_tls_verify   = true
}

Plaintext on which server urls? I'd assume bootstrap ones. You sure you are using zookeeper? As from what I understand the library maps to --bootstrap-server ... it doesn't use --zookeeper.connect=.... Is your cluster public as well?

@qq304635576 I ditched SCRAM and zookeeper. I ended up using IAM Auth which allows you to bypass zookeeper. I used https://github.com/devshawn/kafka-gitops using the following script from this comment to setup IAM auth.

devshawn/kafka-gitops#82 (comment)

[qq304635576]

@arinhouck Actually, Looks like no need to care about zookeeper, I can modify ACL by a client app named "offset explorer 2" without configuring Zookeeper over MSK using SASL_SSL manually. IAM Auth is a new feature, which is owned by AWS MSK only. considering MSK as bus info channel, It should be the most widely compatible with apps for auth. that's why I chose SASL_SSL. moreover, will check your recommendation, maybe I will change to IAM auth in future. thanks.

[qq304635576]

@arinhouck
Good news! I tested again via SASL_SSL & Port:9096. It woks.
(1) Set up as below:

provider "msk" {
bootstrap_servers = var.msk_kafka_brokers
tls_enabled = true
skip_tls_verify = true
sasl_username = local.raw_data.username
sasl_password = local.raw_data.password
sasl_mechanism = "scram-sha512"
}

(2)Terraform output:

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following
symbols:

• create

Terraform will perform the following actions:

##kafka_acl.brokertopic will be created

• resource "kafka_acl" "brokertopic" {
  • acl_host = "*"
  • acl_operation = "All"
  • acl_permission_type = "Allow"
  • acl_principal = "User:broker"
  • id = (known after apply)
  • resource_name = "TEST_"
  • resource_pattern_type_filter = "Prefixed"
  • resource_type = "Topic"
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.

Enter a value: yes

kafka_acl.brokertopic: Creating...
kafka_acl.brokertopic: Creation complete after 2s [id=User:broker|*|All|Allow|Topic|TEST_|Prefixed]
Releasing state lock. This may take a few moments...

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.