[Questions] Upgrade from 3.13.6 to 4.0.6

[MonicaMagoniCom]

Community Support Policy

• I have read RabbitMQ's Community Support Policy
• I run RabbitMQ 4.x, the only series currently covered by community support
• I promise to provide all relevant information (versions, logs from all nodes, rabbitmq-diagnostics output, detailed reproduction steps)

RabbitMQ version used

4.0.6

Erlang version used

27.3.x

Operating system (distribution) used

Container-Optimized OS from Google

How is RabbitMQ deployed?

Kubernetes Operator(s) from Team RabbitMQ

rabbitmq-diagnostics status output

See https://www.rabbitmq.com/docs/cli to learn how to use rabbitmq-diagnostics

# PASTE OUTPUT HERE, BETWEEN BACKTICKS

Logs from node 1 (with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

# PASTE LOG HERE, BETWEEN BACKTICKS

Logs from node 2 (if applicable, with sensitive values edited out)

No response

Logs from node 3 (if applicable, with sensitive values edited out)

No response

rabbitmq.conf

See https://www.rabbitmq.com/docs/configure#config-location to learn how to find rabbitmq.conf file location

# PASTE rabbitmq.conf HERE, BETWEEN BACKTICKS

Steps to deploy RabbitMQ cluster

I have rabbitMQ cluster with version 3.13.6 running with cluster topology operator on Kubernetes.

Steps to reproduce the behavior in question

I updated the RabbitMQCluster from 3.13.6 to 4.0.6 but something went wrong. First of all, the first pod that the operator killed, got stuck on 'Terminating' for about 15minutes; to solve this, I did a
kubectl delete po rabbitmq-server-2 --force

the pod was terminated but when it was restarted (with version 4.0.6), it kept restarting and from the log this is the error I got:

Kernel pid terminated (application_controller) ("{application_start_failure,rabbit,{{{function_clause,[{rabbit_ff_controller,'-list_feature_flags_enabled_somewhere/2-inlined-0-',[classic_queue_mirroring,state_changing,#{classic_mirrored_queue_version => true,classic_queue_type_delivery_support => true,detailed_queues_endpoint => true,direct_exchange_routing_v2 => true,drop_unroutable_metric => true,empty_basic_get_metric => true,feature_flags_v2 => true,implicit_default_bindings => true,listener_records_in_ets => true,maintenance_mode_status => true,message_containers => true,message_containers_deaths_v2 => true,quorum_queue => true,quorum_queue_non_voters => true,restart_streams => true,stream_filtering => true,stream_queue => true,stream_sac_coordinator_unblock_group => true,stream_single_active_consumer => true,stream_update_config_command => true,tracking_records_in_ets => true,user_limits => true,virtual_host_metadata => true,classic_queue_mirroring => true}],[{file,"rabbit_ff_controller.erl"},{line,1

advanced.config

No response

Application code

No response

Kubernetes deployment file

apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
labels:
app: rabbitmq-quorum
name: rabbitmq-quorum
namespace: rabbitmq-system-quorum
spec:
replicas: 3
override:
statefulSet:
spec:
updateStrategy:
rollingUpdate:
partition: 0
image: rabbitmq:4.0.6-management
service:
type: ClusterIP
persistence:
storageClassName: standard
storage: 5Gi
resources:
requests:
cpu: 1500m
memory: 5Gi
limits:
cpu: 2000m
memory: 5Gi
rabbitmq:
additionalPlugins:
- rabbitmq_management
- rabbitmq_peer_discovery_k8s
- rabbitmq_prometheus
additionalConfig: |
cluster_formation.peer_discovery_backend = rabbit_peer_discovery_k8s
cluster_formation.k8s.host = kubernetes.default.svc.cluster.local
cluster_formation.k8s.address_type = hostname
cluster_formation.k8s.service_name = rabbitmq-test
vm_memory_high_watermark.relative = 0.65
cluster_formation.node_cleanup.interval = 10
cluster_formation.discovery_retry_interval = 500
cluster_partition_handling = autoheal
queue_master_locator = min-masters
loopback_users.guest = false
disk_free_limit.absolute = 1GB
advancedConfig: ""

What problem are you trying to solve?

I'm trying to understand why the pod was stuck in Terminating and why it was not able to run with 4.0.6.
I had to recreate the cluster from scratch in order to make it work again