Kafka certificate management with cert-manager

[barryzhounb]

I know we can install our own certificate https://strimzi.io/docs/operators/latest/full/using.html#installing-your-own-ca-certificates-str, however, If we provide our own certificates, we must manually renew them when needed.

After I read an article https://strimzi.io/blog/2021/05/07/deploying-kafka-with-lets-encrypt-certificates/, it describe how listener leverages external certificates from cert-manager, it mentioned certificate renewals will be done automatically by cert-manager and Strimzi. Could I understand this solution address the certificate issue of cluster-ca-cert and cluster-ca?

I am wondering for client-ca-cert and client-ca whether there is similar solution? we expect both client-ca-cert and client-ca can get certificate from cert-manager and certificate renewals can be done automatically by cert-manager and Strimzi. Is it possible? Or Any good point for that?

[scholzj]

The blog post is about the listener certificates: https://strimzi.io/docs/operators/latest/full/using.html#kafka-listener-certificates-str ... Not about custom CAs. These are very different things, so the blog post does not apply in any way to the CA certificates.

[barryzhounb]

Hi @scholzj I thought both listener certificate and cluster-ca-cert were the same, what's diifference from their applied scenario between them?

[scholzj]

The Cluster CA is something used by all cluster components => it is used to generate and handle the internal communication within the cluster. This includes all operators, ZooKeepers, Kafka replication etc. By default, server certificates signed by the Cluster CA are also used for the TLS enabled listeners.

Listener certificate is way how to provide your own TLS server certificate for a specific listener. Such certificate will be used instead of the default one signed by the Cluster CA, but it will be used only by the listener where you configure it. So it will not be used for ZooKeepers, replication etc. It is also easier to use because it does not have to be a CA. So everything around it is easier -> renewals, integration with Let's Encrypt or Cert Manager etc.

[barryzhounb]

Hi @scholzj ,
Suppose listener use certificate that is signed by cert-manager, client uses certificates that is generated from kafka-user to connect to Kafka. When client connects Kafka, listener can verify client via kafka-user's certificates, because kafka-user's CA info have been put in cluster-ca's truststore, here it seems that kafka client doesn't need to know any listener's certificate, client can use Kafka-user's certificate to connect kafka. My question is, why listener need a certificate?

[scholzj]

TBH, I'm not sure I'm the best person to explain how TLS basics works. I guess Google might give you some better guides from people who can explain things better. But basically ...

• Each TLS server needs its server certificate (this is the listener certificate when you configure it, or the certificate generated by Strimzi and signed by the Cluster CA if you don't configure the listener certificates).

• When TLS clients connects to the server, it authenticates it. This is done using the public key which signed the server certificate. If the server certificate is signed by a public CA such as Let's Encrypt, the clients have usually the certificates already bundled somewhere. If the server certificate is signed by some custom CA such as the Strimzi CA, you will need to configure the client and provide it with the public CA key. This basically happens anytime you use encryption.

• Optionally, you can in addition to the server authentication enable also client authentication. In that case, the client has a client certificate and the server has the public key of the CA which signed it. So this time it is the other way around => the server verifies the client.

• In case of Strimzi, the client certificate can be for example generated with the KafkaUser resource. And the Clients CA is the CA used by the broker to verify the clients identity.

[Abdul1929]

Hi schollzj,

I have deployed strimzi kafka and kafka connect with the helm chart. I have used below steps to connect to my kafka cluster and able to produce and consume the msgs.

1. kubectl get secret -n -o jsonpath='{.data.password}' | base64 --decode > user.password
2. kubectl get secret -cluster-ca-cert -n -o jsonpath='{.data.ca.crt}' | base64 --decode > ca.crt
3. kubectl get secret -cluster-ca-cert -n -o jsonpath='{.data.ca.password}' | base64 --decode > ca.password
4. keytool -keystore client-truststore.jks -storetype PKCS12 -alias ca-crt -storepass -keypass -import -file ca.crt -noprompt
5. Prepare the client.properties file (as shown below), which contains the user credentials.
   security.protocol=SASL_SSL
   sasl.mechanism=SCRAM-SHA-512
   ssl.truststore.location=
   ssl.truststore.password=
   sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="" password="<user.password>";
6. ./bin/kafka-console-producer.sh --broker-list : --topic my-topic --producer.config
7. ./bin/kafka-console-consumer.sh --bootstrap-server : --topic my-topic --consumer.config --from-beginning

From these steps I got the below files.
ca.crt , user.password , ca.password, client-truststore.jks,

I want to implement SSL for this kafka connect. I have made the changes in template folder of strimzi kafka connect as below. tls part i have added under spec as per official strimzi doc.

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
name: {{ .Values.cluster.kafkaconnectname }}
spec:
replicas: {{ .Values.replicaCount }}
bootstrapServers: {{ .Values.bootstrapserverIP }}
tls:
trustedCertificates:
- secretName: my-cluster-cluster-ca-cert
certificate: ca.crt
config:
group.id: my-connect-cluster
offset.storage.topic: my-connect-cluster-offsets
config.storage.topic: my-connect-cluster-configs
status.storage.topic: my-connect-cluster-status
config.storage.replication.factor: {{ .Values.replicaCount }}
offset.storage.replication.factor: {{ .Values.replicaCount }}
status.storage.replication.factor: {{ .Values.replicaCount }}
resources:
requests:
cpu: {{ .Values.connectresources.reqcpu }}
memory: {{ .Values.connectresources.reqmemory }}
limits:
cpu: {{ .Values.connectresources.limitcpu }}
memory: {{ .Values.connectresources.limitmemory }}
logging:
type: inline
loggers:
log4j.rootLogger: "INFO"
readinessProbe:
initialDelaySeconds: 15
timeoutSeconds: 5
livenessProbe:
initialDelaySeconds: 15
timeoutSeconds: 5
jvmOptions:
"-Xmx": "1g"
"-Xms": "1g"
image: {{ .Values.connectimage }}
template:
pod:
imagePullSecrets:
- name: regcred-kafka-connect
connectContainer:
env:
- name: JAEGER_SERVICE_NAME
value: my-jaeger-service
- name: JAEGER_AGENT_HOST
value: jaeger-agent-name
- name: JAEGER_AGENT_PORT
value: "6831"

After this point I am stuck.. official docs are bit confusing and after this change in template folder https connection not established.

Can you pls help here

[scholzj]

So, what would the end-users do with that endpoint? You understand that it is for managing connectors etc. and not for receiving or sending messages, right? In general, if you really want to do it, you could create an ingress or front it with some proxy or API Gateway. But the use-case is not really clear to me since you are mixing a very different things together.

[Abdul1929]

kafka template.txt
kafka deployment.txt
kafka connect template.txt
kafka connect deployment.txt

These are my template and deployment file for kafka and kafka connect.

I am getting this error in kafka connect logs.
2021-10-25 16:36:59,606 WARN [AdminClient clientId=adminclient-1] Connection to node -1 (my-cluster-kafka-bootstrap.abdul-test.svc/10.97.137.91:9092) terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue. (org.apache.kafka.clients.NetworkClient) [kafka-admin-client-thread | adminclient-1]

[scholzj]

I don't know what your actual configuration is for the Kafka cluster and Connect cluster. But this error often suggests that you are trying to connect with TLS against non-tls listener.

[Abdul1929]

I have done the deployment with helm chart.. the above one I have attached are both template and deployment file for kafka and kafka connect.

and this one I have created client.properties file for connecting kafka cluster.
security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
ssl.truststore.location=/home/ebarafd/client-truststore.jks
ssl.truststore.password=zi9pl6wPMOTS
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="my-user" password="ibxXdpRVVP8O";

Do I need to add listner property here.. If yes then what should be the value of parameter

[Abdul1929]

Or I need to make any changes in kafka connect deployment as attached kafka-connect template
kafka connect template.txt

[Abdul1929]

I have put secret details (my-cluster-cluster-ca-cert) as below in config file.

kubectl get secrets -n abdul-test
NAME TYPE DATA AGE
default-token-kvdvs kubernetes.io/service-account-token 3 13d
my-cluster-clients-ca Opaque 1 3h35m
my-cluster-clients-ca-cert Opaque 3 3h35m
my-cluster-cluster-ca Opaque 1 3h35m
my-cluster-cluster-ca-cert Opaque 3 3h35m
my-cluster-cluster-operator-certs Opaque 4 3h35m
my-cluster-connect-connect-token-xmb6x kubernetes.io/service-account-token 3 3h29m
my-cluster-entity-operator-certs Opaque 4 3h30m
my-cluster-entity-operator-token-bnrvq kubernetes.io/service-account-token 3 3h30m
my-cluster-kafka-brokers Opaque 4 3h34m
my-cluster-kafka-token-gp7wr kubernetes.io/service-account-token 3 3h34m
my-cluster-zookeeper-nodes Opaque 4 3h35m
my-cluster-zookeeper-token-gts2h kubernetes.io/service-account-token 3 3h35m
my-user Opaque 2 3h30m
regcred-kafka kubernetes.io/dockerconfigjson 1 3h35m
regcred-kafka-connect kubernetes.io/dockerconfigjson 1 3h29m
sh.helm.release.v1.kafka-connect.v1 helm.sh/release.v1 1 3h29m
sh.helm.release.v1.kafka.v1 helm.sh/release.v1 1 3h35m

[Abdul1929]

curl -kv https://10.96.96.132:8083

• About to connect() to 10.96.96.132 port 8083 (#0)
• Trying 10.96.96.132...
• Connected to 10.96.96.132 (10.96.96.132) port 8083 (#0)
• Initializing NSS with certpath: sql:/etc/pki/nssdb
• NSS error -12263 (SSL_ERROR_RX_RECORD_TOO_LONG)
• SSL received a record that exceeded the maximum permissible length.
• Closing connection 0
  curl: (35) SSL received a record that exceeded the maximum permissible length.

[scholzj]

The REST interface does not use TLS. So I don't think this would work. Use HTTP.

[Abdul1929]

I beleive http anyhow will work here..

curl -kv http://10.96.96.132:8083

• About to connect() to 10.96.96.132 port 8083 (#0)
• Trying 10.96.96.132...
• Connected to 10.96.96.132 (10.96.96.132) port 8083 (#0)

GET / HTTP/1.1
User-Agent: curl/7.29.0
Host: 10.96.96.132:8083
Accept: /

< HTTP/1.1 200 OK
< Date: Wed, 20 Oct 2021 12:34:14 GMT
< Content-Type: application/json
< Content-Length: 91
< Server: Jetty(9.4.33.v20201020)
<

• Connection #0 to host 10.96.96.132 left intact
  {"version":"2.7.0","commit":"448719dc99a19793","kafka_cluster_id":"UVwntrKlTcGYUddpc33fig"}[ebarafd@seskp1comp045 templates]$

[Abdul1929]

@scholzj can u suggest here