RabbitMQ Cluster created via RabbitMQ Operator and Istio integration

[lordigon]

Hi everyone,
I created a RabbitMQ cluster using RabbitMQ Operator with this yaml file

apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  name: test-rabbit
spec:
  replicas: 1
  rabbitmq:
    additionalConfig: |
      log.console.level = info
      default_user= guest
      default_pass = guest
      default_user_tags.administrator = true
    additionalPlugins:
      - rabbitmq_mqtt
      - rabbitmq_jms_topic_exchange
      - rabbitmq_web_mqtt
  service:
    type: ClusterIP

I tried to add Istio and configure it but looking at the result via Kiali console some protocol seems not supported automatically.

I'm not an istio expert but reading its documentation Istio automatically manage TCP port using appProtocol property
This is the list of TCP ports used by the RabbitMQ Cluster with appProtocol values

 ports:
  - appProtocol: amqp
    name: amqp
    nodePort: 32204
    port: 5672
    protocol: TCP
    targetPort: 5672
  - appProtocol: http
    name: management
    nodePort: 30513
    port: 15672
    protocol: TCP
    targetPort: 15672
  - appProtocol: mqtt
    name: mqtt
    nodePort: 31327
    port: 1883
    protocol: TCP
    targetPort: 1883

so amqp within appProtocol filed is not recognised as valid Istio supported protocol.
Looking at Istio documentation supported protocol values inside appProtocol field are:

http, http2, https, tcp, pls, grpc, grpc-web, mongo, mysql, redis.

Is that a bug or something is missing me?

Best regards,
Flavio

[lukebakken]

Based on the documentation you should use tcp, NOT amqp

[lordigon]

The problem is RabbitMQ services statefulsets and other resources with their properties (like appProtocol) are created and managed by the operator so they are not under my control...as far as I know.

[Gsantomaggio]

so amqp within appProtocol filed is not recognised as valid Istio supported protocol.

What you see it is just a warning and not an actual problem:
When you enable mTLS:

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
spec:
  mtls:
    mode: STRICT

The Pods are in mTLS, you can test it with openssl s_client -connect <<RABBITMQ-SERVICE>>:5672 for ex:

openssl s_client -connect rabbitmq-prod:5672 
Can't use SSL_get_servername
depth=1 O = cluster.local
verify error:num=19:self signed certificate in certificate chain
verify return:1
depth=1 O = cluster.local
verify return:1
depth=0
verify return:1
140629418255680:error:1409445C:SSL routines:ssl3_read_bytes:tlsv13 alert certificate required:../ssl/record/rec_layer_s3.c:1543:SSL alert number 116
CONNECTED(00000003)
---
Certificate chain
 0 s:
   i:O = cluster.local
 1 s:O = cluster.local
   i:O = cluster.local
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIDTjCCAjagAwIBAgIRAMG+5j/JZ8s+3b+XDQ7YFkwwDQYJKoZIhvcNAQELBQAw
GDEWMBQGA1UEChMNY2x1c3Rlci5sb2NhbDAeFw0yMjA3MTQyMjQ3NTJaFw0yMjA3
MTUyMjQ5NTJaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCv52Iy
qlMP3I0kKjjX35bbv4tRZZjo0rBEMFMDvPBDd4w0/6LNxHhmpAz06OU+YTvCjZm
..........

you can easily test what happens if you disable the mTLS :

openssl s_client -connect rabbitmq-prod:5672
CONNECTED(00000003)
140619414811968:error:1408F10B:SSL routines:ssl3_get_record:wrong version number:../ssl/record/ssl3_record.c:331:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 5 bytes and written 283 bytes

As you can see, the pod is not in TLS anymore.

Same for the Management UI:

 curl -kv  https://rabbitmq-prod:5672
*   Trying 10.116.20.119:5672...
* TCP_NODELAY set
* Connected to rabbitmq-prod (10.116.20.119) port 5672 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* error:1408F10B:SSL routines:ssl3_get_record:wrong version number
* Closing connection 0
curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number
istio-proxy@rabbitmq-prod-server-0:/$ curl -kv  https://rabbitmq-prod:5672
*   Trying 10.116.20.119:5672...
* TCP_NODELAY set
* Connected to rabbitmq-prod (10.116.20.119) port 5672 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: [NONE]
*  start date: Jul 15 17:57:22 2022 GMT
*  expire date: Jul 16 17:59:22 2022 GMT
*  issuer: O=cluster.local
*  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* OpenSSL SSL_write: Broken pipe, errno 32
* Failed sending HTTP2 data

[bkoncz]

@lordigon Did you ever manage to solve this?

[kxiong0]

In case anyone is still looking, you can overwrite the appPortocols similar to how the additional ports example creates new ports.
Something like this worked for me:

spec:
  replicas: 3
  override:
    service:
      spec:
        ports:
          - name: prometheus
            protocol: TCP
            port: 15692
            targetPort: 15692
            appProtocol: tcp

[ramazan-biyik]

Hi @KXiong2001 did you try to apply 5672 port? I tried to add additional port to service that 5673 will direct to 5672 on port. However istio dont understand that.

[kxiong0]

Hi @KXiong2001 did you try to apply 5672 port? I tried to add additional port to service that 5673 will direct to 5672 on port. However istio dont understand that.

I used the 5672 port. I remember the error message in Kiali worked for 5672 when I changed the port name.
But seems like using port names in Istio standards or not didn't make a huge difference feature wise in my case, so I just left the port name as default.

[mkuratczyk]

Is this a cluster-operator bug that you have to override the appProtocol or is it something specific to Istio? If it is a bug, overriding it is a good workaround, but it should be fixed in the cluster operator.

…

On Fri, Nov 10, 2023 at 9:05 AM K Xiong ***@***.***> wrote: In case anyone is still looking, you can overwrite the appPortocols similar to how the additional ports example <https://github.com/rabbitmq/cluster-operator/blob/main/docs/examples/additionalPorts/rabbitmq.yaml> creates new ports. Something like this worked for me: spec: replicas: 3 override: service: spec: ports: - name: prometheus protocol: TCP port: 15692 targetPort: 15692 appProtocol: tcp — Reply to this email directly, view it on GitHub <#5226 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACI7PIWYFAALBE5NAMKHIWLYDXN5PAVCNFSM53RPUVUKU5DIOJSWCZC7NNSXTOKENFZWG5LTONUW63SDN5WW2ZLOOQ5TONJTGA4DCMA> . You are receiving this because you are subscribed to this thread.Message ID: <rabbitmq/rabbitmq-server/repo-discussions/5226/comments/7530810@ github.com>

-- Michał RabbitMQ team

[kxiong0]

This is specific to Istio. Like lordigon mentioned, Istio doesn’t support some appProtocols, and those include ones that RabbitMQ operator uses.
I’m fairly new to Istio and RabbitMQ, but I think a workaround should be sufficient for this case.