Handshake issue with Broker | Ingress | 0.31.1 #7407

senthil13 · 2022-09-28T22:27:16Z

senthil13
Sep 28, 2022

Deployed latest strimzi and able to access inside the k8s cluster but not outside the cluster when we enable TLS. We are getting SSL Handshake failure. I was following the link and used the below command to generate the connecting truststore for the console producer.

kubectl get secret cluster-name-cluster-ca-cert -o jsonpath='{.data.ca\.crt}' | base64 -d > ca.crt
keytool -import -trustcacerts -alias root -file ca.crt -keystore truststore.jks -storepass password -noprompt

When I see a cert with the open ssl - it shows a bad certificate.

openssl s_client -connect kafka-bootstrap.tcp.XXXX.XXX.XXX.com:443 -showcerts
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 19 (self signed certificate in certificate chain)
---
140291563337024:error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate:../ssl/record/rec_layer_s3.c:1543:SSL alert number 42

Deployment files that we used.

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: pingkafka-cluster-psingle
spec:
  kafka:
    template:
      pod:
        schedulerName: stork
    version: 3.2.3
    replicas: 1
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
      - name: external
        port: 9094
        type: ingress
        tls: true
        authentication:
          type: tls
        configuration:
          bootstrap:
            host: kafka-bootstrap.tcp.XXX.XXX.XXX.com
            annotations:
              external-dns.alpha.kubernetes.io/hostname: kafka-bootstrap.tcp.XXX.XXX.XXX.com.
              external-dns.alpha.kubernetes.io/ttl: "60"
          brokers:
          - broker: 0
            host: kafka-broker.tcp.XXX.XXX.XXX.com
            annotations:
              external-dns.alpha.kubernetes.io/hostname: kafka-broker.tcp.XXX.XXX.XXX.com.
              external-dns.alpha.kubernetes.io/ttl: "60"     
    config:
      offsets.topic.replication.factor: 1
      transaction.state.log.replication.factor: 1
      transaction.state.log.min.isr: 1
      default.replication.factor: 1
      min.insync.replicas: 1
      inter.broker.protocol.version: "3.2"
      num.partitions: 1

Logs



[2022-09-28 18:04:56,710] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2022-09-28 18:04:57,159] DEBUG Created SSL context with keystore null, truststore SecurityStore(path=./truststore.jks, modificationTime=Wed Sep 28 16:50:48 EDT 2022), provider SunJSSE. (org.apache.kafka.common.security.ssl.SslEngineBuilder)
[2022-09-28 18:04:57,200] DEBUG [Producer clientId=console-producer] Starting Kafka producer I/O thread. (org.apache.kafka.clients.producer.internals.Sender)
[2022-09-28 18:04:57,202] DEBUG [Producer clientId=console-producer] Initialize connection to node kafka-bootstrap.tcp.devops.XXX.XXXc.com:443 (id: -1 rack: null) for sending metadata request (org.apache.kafka.clients.NetworkClient)
[2022-09-28 18:04:57,205] DEBUG [Producer clientId=console-producer] Initiating connection to node kafka-bootstrap.tcp.devops.XXX.XXXc.com:443 (id: -1 rack: null) using address kafka-bootstrap.tcp.devops.XXX.XXXc.com/XXX.XX.XX.XXX (org.apache.kafka.clients.NetworkClient)
[2022-09-28 18:04:57,208] INFO Kafka version: 5.5.10-ce (org.apache.kafka.common.utils.AppInfoParser)
[2022-09-28 18:04:57,208] INFO Kafka commitId: 43102f948475ddc3 (org.apache.kafka.common.utils.AppInfoParser)
[2022-09-28 18:04:57,208] INFO Kafka startTimeMs: 1664402697200 (org.apache.kafka.common.utils.AppInfoParser)
[2022-09-28 18:04:57,211] DEBUG [Producer clientId=console-producer] Kafka producer started (org.apache.kafka.clients.producer.KafkaProducer)
>[2022-09-28 18:04:57,260] DEBUG [Producer clientId=console-producer] Created socket with SO_RCVBUF = 32768, SO_SNDBUF = 102400, SO_TIMEOUT = 0 to node -1 (org.apache.kafka.common.network.Selector)
[2022-09-28 18:04:57,578] DEBUG [Producer clientId=console-producer] Completed connection to node -1. Fetching API versions. (org.apache.kafka.clients.NetworkClient)
[2022-09-28 18:04:57,707] INFO [Producer clientId=console-producer] Failed authentication with kafka-bootstrap.tcp.devops.XXX.XXXc.com/XXX.XX.XX.XXX (SSL handshake failed) (org.apache.kafka.common.network.Selector)
[2022-09-28 18:04:57,709] DEBUG [Producer clientId=console-producer] Node -1 disconnected. (org.apache.kafka.clients.NetworkClient)
[2022-09-28 18:04:57,711] ERROR [Producer clientId=console-producer] Connection to node -1 (kafka-bootstrap.tcp.devops.XXX.XXXc.com/XXXXXXXXXXXXX:443) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)
[2022-09-28 18:04:57,711] WARN [Producer clientId=console-producer] Bootstrap broker kafka-bootstrap.tcp.devops.XXX.XXXc.com:443 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2022-09-28 18:04:57,810] DEBUG [Producer clientId=console-producer] Initialize connection to node kafka-bootstrap.tcp.devops.XXX.XXXc.com:443 (id: -1 rack: null) for sending metadata request (org.apache.kafka.clients.NetworkClient)
[2022-09-28 18:04:57,810] DEBUG [Producer clientId=console-producer] Initiating connection to node kafka-bootstrap.tcp.devops.XXX.XXXc.com:443 (id: -1 rack: null) using address kafka-bootstrap.tcp.devops.XXX.XXXc.com/XXX.XX.XX.XXX (org.apache.kafka.clients.NetworkClient)
[2022-09-28 18:04:57,835] DEBUG [Producer clientId=console-producer] Created socket with SO_RCVBUF = 32768, SO_SNDBUF = 102400, SO_TIMEOUT = 0 to node -1 (org.apache.kafka.common.network.Selector)
[2022-09-28 18:04:57,838] DEBUG [Producer clientId=console-producer] Completed connection to node -1. Fetching API versions. (org.apache.kafka.clients.NetworkClient)
[2022-09-28 18:04:57,920] INFO [Producer clientId=console-producer] Failed authentication with kafka-bootstrap.tcp.devops.XXX.XXXc.com/XXX.XX.XX.XXX (SSL handshake failed) (org.apache.kafka.common.network.Selector)
[2022-09-28 18:04:57,920] DEBUG [Producer clientId=console-producer] Node -1 disconnected. (org.apache.kafka.clients.NetworkClient)
[2022-09-28 18:04:57,920] ERROR [Producer clientId=console-producer] Connection to node -1 (kafka-bootstrap.tcp.devops.XXX.XXXc.com/XXXXXXXXXXXXX:443) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)
[2022-09-28 18:04:57,920] WARN [Producer clientId=console-producer] Bootstrap broker kafka-bootstrap.tcp.devops.XXX.XXXc.com:443 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2022-09-28 18:04:58,021] DEBUG [Producer clientId=console-producer] Give up sending metadata request since no node is available (org.apache.kafka.clients.NetworkClient)
[2022-09-28 18:04:58,072] DEBUG [Producer clientId=console-producer] Give up sending metadata request since no node is available (org.apache.kafka.clients.NetworkClient)
[2022-09-28 18:04:58,123] DEBUG [Producer clientId=console-producer] Give up sending metadata request since no node is available (org.apache.kafka.clients.NetworkClient)
[2022-09-28 18:04:58,174] DEBUG [Producer clientId=console-producer] Give up sending metadata request since no node is available (org.apache.kafka.clients.NetworkClient)
[2022-09-28 18:04:58,224] DEBUG [Producer clientId=console-producer] Give up sending metadata request since no node is available (org.apache.kafka.clients.NetworkClient)
[2022-09-28 18:04:58,275] DEBUG [Producer clientId=console-producer] Initialize connection to node kafka-bootstrap.tcp.devops.XXX.XXXc.com:443 (id: -1 rack: null) for sending metadata request (org.apache.kafka.clients.NetworkClient)
[2022-09-28 18:04:58,275] DEBUG [Producer clientId=console-producer] Initiating connection to node kafka-bootstrap.tcp.devops.XXX.XXXc.com:443 (id: -1 rack: null) using address kafka-bootstrap.tcp.devops.XXX.XXXc.com/XXX.XX.XX.XXX (org.apache.kafka.clients.NetworkClient)

scholzj · 2022-09-28T22:36:35Z

scholzj
Sep 28, 2022
Maintainer

You do not share the configuration of your client. Typically, these issues will be one of:

TLS Passthrough is not enabled in your Ingress controller
You did not configure your client for TLS client authentication which you have configured in your Kafka CR (i.e. you would need to create a user and pass the keystroe with theuser certificate to the Kafka client as well)

So check for these two -> it might be that you have both of them at the same time of course.

The OpenSSL command is missing the -servername option to handle the TLS-SNI. When you fix that and run it also with -showcerts` option, it should help you to see about the first issue.

For the second issue, you probably know how you configured the Kafka client and if you created a user.

5 replies

senthil13 Sep 28, 2022
Author

The client is just a console producer with the below command. While connecting to the external Do I need to connect with the username and password? I didn't create one while creating the kafka cluster. I used the example - examples/kafka/kafka-persistent-single.yaml.

kubectl get secret kafka-cluster-psingle-cluster-ca-cert -n ping-kafka -o jsonpath='{.data.ca\.crt}' | base64 -d > ca.crt
keytool -import -trustcacerts -alias root -file ca.crt -keystore truststore.jks -storepass password -noprompt
Enable debug on the console producer
kafka-console-producer --broker-list kafka-bootstrap.tcp.devops.XXX.XXXX.com:443 --producer-property security.protocol=SSL --producer-property ssl.truststore.password=password --producer-property ssl.truststore.location=./truststore.jks --topic my-topic

included the servername in the openssl command and the bad certificate error is gone.

openssl s_client -connect kafka-bootstrap.tcp.devops.XXX.XXX.com:443 --servername -showcerts
CONNECTED(00000003)
depth=0 O = Acme Co, CN = Kubernetes Ingress Controller Fake Certificate
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 O = Acme Co, CN = Kubernetes Ingress Controller Fake Certificate
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
 0 s:O = Acme Co, CN = Kubernetes Ingress Controller Fake Certificate
   i:O = Acme Co, CN = Kubernetes Ingress Controller Fake Certificate
---
Server certificate
-----BEGIN CERTIFICATE-----
<removed cer content>
-----END CERTIFICATE-----
subject=O = Acme Co, CN = Kubernetes Ingress Controller Fake Certificate

issuer=O = Acme Co, CN = Kubernetes Ingress Controller Fake Certificate

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 1443 bytes and written 382 bytes
Verification error: unable to verify the first certificate
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 21 (unable to verify the first certificate)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
    Session-ID: BB197CF758E42B71C5F0F423136C1DE3199BCC3D2F4067C10EC1163B59F7A00A
    Session-ID-ctx:
    Resumption PSK: 0F90E7844F568FADB4D4E4B829C2A0B1148012DDCA4829C302094083A4F6795F7D33FA0F97377A74ABDC33FBB43B5F27
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 600 (seconds)
    TLS session ticket:
    0000 - ed 8a 80 2e 49 73 84 19-10 43 a5 93 7e 48 64 ac   ....Is...C..~Hd.
    0010 - 15 d3 24 89 ad 90 ab a5-59 40 08 20 24 6a f8 20   ..$.....Y@. $j.

    Start Time: 1664404822
    Timeout   : 7200 (sec)
    Verify return code: 21 (unable to verify the first certificate)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
    Session-ID: 455D0C33033B9C5A7B09FBBB175F8C7F3B66ECD3B526A1B9D9353E1067D15338
    Session-ID-ctx:
    Resumption PSK: 128CFD979B8264E740CC285F2DA572199F5B712A6BA23E8BB84CE3D324FB4BA0CF996BF3B4A8D62C9DD8FB860509CB2E
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 600 (seconds)
    TLS session ticket:
    0000 - 79 52 0b 04 ab 52 4c a4-90 3d ee 02 e5 d7 be f8   yR...RL..=......
    0010 - 09 fd b0 e6 b5 10 39 e7-eb 88 9a 4c 7d 77 aa 1b   ......9....L}w..

    Start Time: 1664404822
    Timeout   : 7200 (sec)
    Verify return code: 21 (unable to verify the first certificate)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK

^C

scholzj Sep 28, 2022
Maintainer

The client is just a console producer with the below command. While connecting to the external Do I need to connect with the username and password? I didn't create one while creating the kafka cluster. I used the example - examples/kafka/kafka-persistent-single.yaml.

Right, so as I said, you did not configured the user authentication / TLS keystore. This part of the Kafka CR:

        authentication:
          type: tls

Says that you want to use authentication using TLS client certificates. So you have to configure one in the client (you can use the User Operator and the KafkaUser CR to create the user). Or you need to remove this from the Kafka CR and then you do not need any authentication.

The option is AFAIK -servername (single -). It also needs to have an value which is the address where you connect (e.g. kafka-bootstrap.tcp.XXXX.XXX.XXX.com). If after setting that in the OpenSSL command you still get Kubernetes Ingress Controller Fake Certificate, then it means your Ingress controller does not have TLS passthrough enabled.

senthil13 Sep 28, 2022
Author

Hi - what is Kafka CR stands far?

As far as I can tell this is the first time we are setting up the Kafka using the strimzi so excuse me if I asked any wrong questions here. As I mentioned earlier I didn't create any username and password do I need to do? where d I need to enable the authentication - type: TLS while creating the cluster operator or while creating the Kafka and the zookeeper. The document link - https://strimzi.io/blog/2019/05/23/accessing-kafka-part-5/ - is not mentioned any user name or password. So I am thinking of Kafka using here plain and SSL(TLS), So inside the k8s cluster, we can connect with plain 9092 fine no issue there. But from the outside connection is having issues, If I put type is ingress or load balancer without tls: true I am unable to bring up the Kafka pod.

I used the proper OpenSSL command, now I am getting the bad cert at the bottom.

openssl s_client -connect kafka-bootstrap.tcp.devops.XXX.XXX.com:443 -servername kafka-bootstrap.tcp.devops.XXX.XXXXc.com -showcerts
CONNECTED(00000003)
depth=1 O = io.strimzi, CN = cluster-ca v0
verify error:num=19:self signed certificate in certificate chain
verify return:1
depth=1 O = io.strimzi, CN = cluster-ca v0
verify return:1
depth=0 O = io.strimzi, CN = kafka-cluster-psingle-kafka
verify return:1
---
Certificate chain
 0 s:O = io.strimzi, CN = kafka-cluster-psingle-kafka
   i:O = io.strimzi, CN = cluster-ca v0
-----BEGIN CERTIFICATE-----
<Removed cert content>
-----END CERTIFICATE-----
 1 s:O = io.strimzi, CN = cluster-ca v0
   i:O = io.strimzi, CN = cluster-ca v0
-----BEGIN CERTIFICATE-----
<Removed cert content>
-----END CERTIFICATE-----
---
Server certificate
subject=O = io.strimzi, CN = kafka-cluster-psingle-kafka

issuer=O = io.strimzi, CN = cluster-ca v0

---
Acceptable client certificate CA names
O = io.strimzi, CN = clients-ca v0
Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA1:RSA+SHA1
Shared Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 3760 bytes and written 441 bytes
Verification error: self signed certificate in certificate chain
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 19 (self signed certificate in certificate chain)
---
139732994987328:error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate:../ssl/record/rec_layer_s3.c:1543:SSL alert number 42

Is it possible to enable external access without TLS for external type ingress or load balancer

Also, I see the passthrough enabled on my Ingress

kubectl describe ingress/kafka-cluster-psingle-kafka-0 -n kafka | grep ssl-passthroug
                                        ingress.kubernetes.io/ssl-passthrough: true
                                        nginx.ingress.kubernetes.io/ssl-passthrough: true
kubectl describe ingress/kafka-cluster-psingle-kafka-bootstrap -n kafka | grep ssl-passthroug
                                           ingress.kubernetes.io/ssl-passthrough: true
                                           nginx.ingress.kubernetes.io/ssl-passthrough: true

Thanks

scholzj Sep 28, 2022
Maintainer

what is Kafka CR stands far?

CR stands for custom resource. That is the resource you used to deploy the Kafka cluster. That is basically this YAML:

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: pingkafka-cluster-psingle
spec:
  # ...

As I mentioned earlier I didn't create any username and password do I need to do?

It is up to you what kind of security you want to use. You can enable authentication or disable it, choose the type (whether you use TLS certificates or username and password etc.). Right now, the Kafka CR you shared above has this section in it:

        authentication:
          type: tls

Which tells the operator to configure the broker to require TLS client authentication (mTLS) fro the clients connecting using the Ingress connection. Maybe the best way to start is to remove it first, get everything working without it and then you can do and add authentication as a next step.

So either:

Edit the YAML in some file, remove the snippet above, save it and run kubectl apply -f <filename>
Or do kubectl edit kafka pingkafka-cluster-psingle, remove it in the text editor it opens, save the changes and exit.

The document link - https://strimzi.io/blog/2019/05/23/accessing-kafka-part-5/ - is not mentioned any user name or password.

It does not. But it also doesn't have the authentication part mentioned above which you have there.

But from the outside connection is having issues, If I put type is ingress or load balancer without tls: true I am unable to bring up the Kafka pod.

The tls flag enabeld or disabled TLS encryption. It is mandatory, so you always need to have it there set either to false or true. The Ingress listener needs to always have it set to true because of how Ingress works. With load balancer listener, you can set tls: false and connect without TLS encryption if you want as you do on port 9092 from inside the Kafka cluster.

I used the proper OpenSSL command, now I am getting the bad cert at the bottom.

So, the output suggests your Ingress works fine and has TLS passthrough enabled. So that is good. The reason why the OpenSSL gives you the error at the end is again because of the TLS client authentication you have enabled. But the Ingress is good and I think that if you disable the authentication as described above, you Kafka client should start working.

senthil13 Sep 29, 2022
Author

Thanks for the explanation, Issue is resolved. The problem was the two lines. Type ingress just needs the tls:true only, not the following two lines.


        authentication:
          type: tls

If I put those two lines then I need to connect with the user name and password and I removed it and created the pods again. Now I am able to post messages as well as consume it.

Thanks for the help @scholzj , I really appreciate your swift response to my queries. I will accept your last response as an answer.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Strimzi

Handshake issue with Broker | Ingress | 0.31.1 #7407

{{title}}

Replies: 1 comment 5 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Strimzi

Handshake issue with Broker | Ingress | 0.31.1 #7407

senthil13 Sep 28, 2022

Replies: 1 comment · 5 replies

scholzj Sep 28, 2022 Maintainer

senthil13 Sep 28, 2022 Author

scholzj Sep 28, 2022 Maintainer

senthil13 Sep 28, 2022 Author

scholzj Sep 28, 2022 Maintainer

senthil13 Sep 29, 2022 Author

senthil13
Sep 28, 2022

Replies: 1 comment 5 replies

scholzj
Sep 28, 2022
Maintainer

senthil13 Sep 28, 2022
Author

scholzj Sep 28, 2022
Maintainer

senthil13 Sep 28, 2022
Author

scholzj Sep 28, 2022
Maintainer

senthil13 Sep 29, 2022
Author