Peers cannot be discovered when TLS inter-node communication is enabled in Container - 4.0.7-management

[ahi234]

Community Support Policy

• I have read RabbitMQ's Community Support Policy
• I run RabbitMQ 4.x, the only series currently covered by community support
• I promise to provide all relevant information (versions, logs from all nodes, rabbitmq-diagnostics output, detailed reproduction steps)

RabbitMQ version used

4.0.5

Erlang version used

27.2.x

Operating system (distribution) used

Linux (Kubernetes)

How is RabbitMQ deployed?

Community Docker image

rabbitmq-diagnostics status output

Error: unable to perform an operation on node '[email protected]'. Please see diagnostics information and suggestions below.

Most common reasons for this are:

• Target node is unreachable (e.g. due to hostname resolution, TCP connection or firewall issues)
• CLI tool fails to authenticate with the server (e.g. due to CLI tool's Erlang cookie not matching that of the server)
• Target node is not running

In addition to the diagnostics info below:

• See the CLI, clustering and networking guides on https://rabbitmq.com/documentation.html to learn more
• Consult server logs on node [email protected]
• If target node is configured to use long node names, don't forget to use --longnames with CLI tools

DIAGNOSTICS

attempted to contact: ['[email protected]']

[email protected]:

• connected to epmd (port 4369) on rmq-cluster-server-3.rmq-cluster-nodes.rmq
• epmd reports node 'rabbit' uses port 25672 for inter-node and CLI tool traffic
• TCP connection succeeded but Erlang distribution failed
• suggestion: check if the Erlang cookie is identical for all server nodes and CLI tools
• suggestion: check if all server nodes and CLI tools use consistent hostnames when addressing each other
• suggestion: check if inter-node connections may be configured to use TLS. If so, all nodes and CLI tools must do that
• suggestion: see the CLI, clustering and networking guides on https://rabbitmq.com/documentation.html to learn more

Current node details:

• node name: '[email protected]'
• effective user's home directory: /var/lib/rabbitmq
• Erlang cookie hash: xV2OI4mpcYmVE+jVnr58Fg==

Logs from node 1 (with sensitive values edited out)

�[38;5;246m2025-02-28 11:18:19.931963+00:00 [debug] <0.238.0> Peer discovery: backend returned the following configuration:�[0m
�[38;5;246m2025-02-28 11:18:19.931963+00:00 [debug] <0.238.0> {ok,{['[email protected]',�[0m
�[38;5;246m2025-02-28 11:18:19.931963+00:00 [debug] <0.238.0> '[email protected]',�[0m
�[38;5;246m2025-02-28 11:18:19.931963+00:00 [debug] <0.238.0> '[email protected]',�[0m
�[38;5;246m2025-02-28 11:18:19.931963+00:00 [debug] <0.238.0> '[email protected]',�[0m
�[38;5;246m2025-02-28 11:18:19.931963+00:00 [debug] <0.238.0> '[email protected]'],�[0m
�[38;5;246m2025-02-28 11:18:19.931963+00:00 [debug] <0.238.0> disc}}�[0m
�[38;5;246m2025-02-28 11:18:19.932187+00:00 [debug] <0.238.0> Peer discovery: peer node arguments: #{args =>�[0m
�[38;5;246m2025-02-28 11:18:19.932187+00:00 [debug] <0.238.0> ["-ssl_dist_optfile",�[0m
�[38;5;246m2025-02-28 11:18:19.932187+00:00 [debug] <0.238.0> "/etc/rabbitmq/inter_node_tls.config",�[0m
�[38;5;246m2025-02-28 11:18:19.932187+00:00 [debug] <0.238.0> "-proto_dist","inet_tls","-pa",�[0m
�[38;5;246m2025-02-28 11:18:19.932187+00:00 [debug] <0.238.0> "/opt/erlang/lib/erlang/lib/ssl-11.2.8/ebin",�[0m
�[38;5;246m2025-02-28 11:18:19.932187+00:00 [debug] <0.238.0> "-boot","start_sasl","-hidden"],�[0m
�[38;5;246m2025-02-28 11:18:19.932187+00:00 [debug] <0.238.0> name => "rabbit-773-11",�[0m
�[38;5;246m2025-02-28 11:18:19.932187+00:00 [debug] <0.238.0> host =>�[0m
�[38;5;246m2025-02-28 11:18:19.932187+00:00 [debug] <0.238.0> "rmq-cluster-server-4.rmq-cluster-nodes.rmq",�[0m
�[38;5;246m2025-02-28 11:18:19.932187+00:00 [debug] <0.238.0> connection => standard_io,�[0m
�[38;5;246m2025-02-28 11:18:19.932187+00:00 [debug] <0.238.0> wait_boot => infinity,�[0m
�[38;5;246m2025-02-28 11:18:19.932187+00:00 [debug] <0.238.0> longnames => true}�[0m
�[38;5;246m2025-02-28 11:18:20.166467+00:00 [debug] <0.238.0> Peer discovery: using temporary hidden node '[email protected]' to query discovered peers properties�[0m
=PROGRESS REPORT==== 28-Feb-2025::11:18:20.171309 ===
supervisor: {local,inet_gethost_native_sup}
started: [{pid,<0.106.0>},{mfa,{inet_gethost_native,init,[[]]}}]

=PROGRESS REPORT==== 28-Feb-2025::11:18:20.176411 ===
supervisor: {local,kernel_safe_sup}
started: [{pid,<0.105.0>},
{id,inet_gethost_native_sup},
{mfargs,{inet_gethost_native,start_link,[]}},
{restart_type,temporary},
{significant,false},
{shutdown,1000},
{child_type,worker}]

Logs from node 2 (if applicable, with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

# PASTE LOG HERE, BETWEEN BACKTICKS

Logs from node 3 (if applicable, with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

# PASTE LOG HERE, BETWEEN BACKTICKS

rabbitmq.conf

See https://www.rabbitmq.com/docs/configure#config-location to learn how to find rabbitmq.conf file location

# PASTE rabbitmq.conf HERE, BETWEEN BACKTICKS

Steps to deploy RabbitMQ cluster

Following this guide: https://www.rabbitmq.com/docs/clustering-ssl#linux-strategy-two

Definition of the rabbitmq-env file and mounting the SSL files and inter node tls config in operator. Operator starts cluster and in each pod the error will be thrown

Steps to reproduce the behavior in question

https://www.rabbitmq.com/docs/clustering-ssl#linux-strategy-two

advanced.config

See https://www.rabbitmq.com/docs/configure#config-location to learn how to find advanced.config file location

# PASTE advanced.config HERE, BETWEEN BACKTICKS

Application code

# PASTE CODE HERE, BETWEEN BACKTICKS

Kubernetes deployment file

# Relevant parts of K8S deployment that demonstrate how RabbitMQ is deployed
# PASTE YAML HERE, BETWEEN BACKTICKS

What problem are you trying to solve?

I'm attempting to implement TLS for inter-node communication within my RabbitMQ cluster to ensure secure communication between nodes. To achieve this, I've defined the rabbitmq_env.conf file in the RabbitMQ operator and mounted the necessary SSL files along with the inter_node_tls.config file. However, I'm encountering an error on all 5 nodes, indicating that they are using a temporary hidden node and are unable to form a cluster.

In my inter_node_cluster.config file, I've only defined the server configuration section, as my primary goal is to encrypt the communication between nodes. I'm also trying to avoid explicitly defining the specific nodes in my rabbitmq.conf file to form the cluster.

[lukebakken]

Carefully compare what you are doing to my working project here:

https://github.com/lukebakken/docker-rabbitmq-cluster/tree/tls

After cloning, run git submodule update --init to initialize the tls-gen directory, which points here: https://github.com/rabbitmq/tls-gen

The following command will start a cluster with inter-node TLS using rabbitmq:4-management. I just checked that it work successfully. It does use pre-defined node names in rabbitmq.conf.

make DOCKER_FRESH=true up

You need to provide all of the information about how your environment is being configured.

[ahi234]

Thanks a lot, I will check your repo.

However, is there any possibility to not define the other node names in the configuration file ?

Because in my environment, I am using the rabbitmq operator in kubernetes.

[mkuratczyk]

We have an example for setting up TLS between nodes using the Operator:
https://github.com/rabbitmq/cluster-operator/tree/main/docs/examples/mtls-inter-node

I don't see where/why you need to list all the nodes. The example provided only lists them in the TLS cert SANs.

By the way, you might be interested in the new Kubernetes peer discovery mechanism for RabbitMQ 4.1: #13207. However, it's not necessary to set up TLS between nodes, just something that might interest you. We would also appreciate feedback if it works for you.