[Questions] 3.13.x: Switching to OAuth 2 causes nodes to be OOM-killed by Kubernetes #13477
-
Community Support Policy
RabbitMQ version used4.0.7 Erlang version used27.2.x Operating system (distribution) usedKubernetes (AWS EKS Bottlerocket) How is RabbitMQ deployed?Bitnami Helm chart rabbitmq-diagnostics status outputSee https://www.rabbitmq.com/docs/cli to learn how to use rabbitmq-diagnostics Logs from node 1 (with sensitive values edited out)See https://www.rabbitmq.com/docs/logging to learn how to collect logs Logs from node 2 (if applicable, with sensitive values edited out)See https://www.rabbitmq.com/docs/logging to learn how to collect logs Logs from node 3 (if applicable, with sensitive values edited out)See https://www.rabbitmq.com/docs/logging to learn how to collect logs rabbitmq.confSee https://www.rabbitmq.com/docs/configure#config-location to learn how to find rabbitmq.conf file location Steps to deploy RabbitMQ clusterWe're using an ArgoCD apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: 'rabbitmq-stage-sa-east-1'
namespace: 'argocd'
annotations:
argocd.argoproj.io/sync-wave: '50'
labels:
clusterName: company-stage-sa-east-1-k8s
env: stage
region: sa-east-1
vendor: aws
spec:
project: 'addons-stage'
sources:
- repoURL: 'registry-1.docker.io/bitnamicharts'
targetRevision: '4.4.6'
chart: 'rabbitmq-cluster-operator'
helm:
releaseName: 'rabbitmq'
ignoreMissingValueFiles: true
valueFiles:
- '$values/k8s/addons/stage/sa-east-1/rabbitmq.yaml'
- repoURL: 'https://github.com/company/IaC.git'
targetRevision: HEAD
ref: values
destination:
name: 'company-stage-sa-east-1-k8s'
namespace: 'rabbitmq-system'
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=trueAnother apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
name: rabbitmq-cluster
namespace: rabbitmq-system
annotations:
argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
spec:
replicas: 1
persistence:
storageClassName: gp3
storage: 10Gi
resources:
requests:
cpu: 1
memory: 4Gi
limits:
cpu: ''
memory: 4Gi
rabbitmq:
additionalConfig: |
log.console.level = debug
auth_backends.1 = rabbit_auth_backend_oauth2
auth_backends.2 = internal
auth_oauth2.resource_server_id = rabbitmq
auth_oauth2.issuer = https://argocd.k8s.eu-north-1.ops.company.com/api/dex
management.oauth_enabled = true
management.oauth_client_id = rabbitmq
management.oauth_client_secret = <client-secret>
management.oauth_scopes = openid profile email groups
vm_memory_high_watermark.absolute = 3500Mi
advancedConfig: |
[
{rabbitmq_auth_backend_oauth2,
[{scope_prefix, <<>>}
,{extra_scopes_source, <<"groups">>}
,{preferred_username_claims, [<<"email">>, <<"username">>, <<"user_name">>, <<"sub">>]}
,{scope_aliases, #{<<"company:platform">> => [<<"tag:administrator">>, <<"configure:*/*/*">>, <<"write:*/*/*">>, <<"read:*/*/*">>]}}
]
}
].
additionalPlugins:
- rabbitmq_auth_backend_oauth2Steps to reproduce the behavior in questionConfigure OIDC auth and try to login. The pod will consume all memory and be OOMKilled and the browser console will display the below error: 
advanced.configSee https://www.rabbitmq.com/docs/configure#config-location to learn how to find advanced.config file location Application code# PASTE CODE HERE, BETWEEN BACKTICKSKubernetes deployment file# Relevant parts of K8S deployment that demonstrate how RabbitMQ is deployed
# PASTE YAML HERE, BETWEEN BACKTICKSWhat problem are you trying to solve?We're trying to use Dex (the ArgoCD bundled one) with RabbitMQ so that we can manage users via Github. Note that Dex is working as expected for both ArgoCD and the Kubernetes API (via |
Beta Was this translation helpful? Give feedback.
Replies: 4 comments 1 reply
-
|
These are the relevant bits of the Dex config: connectors:
- type: github
id: github
name: GitHub
config:
clientID: "$argocd-github-oauth:clientID"
clientSecret: "$argocd-github-oauth:clientSecret"
orgs:
- name: company
teams:
- engineering
- platform
staticClients:
- id: kubernetes
name: kubernetes
secret: "$dex-client-secret:clientSecret"
redirectURIs:
- "http://localhost:8000"
- id: rabbitmq
name: rabbitmq
secret: "$dex-client-secret:clientSecret"
redirectURIs:
- "https://rabbitmq.k8s.sa-east-1.stage.company.com/js/oidc-oauth/login-callback.html"
web:
allowedOrigins: ['*'] |
Beta Was this translation helpful? Give feedback.
-
|
I noticed that despite using the latest chart 
|
Beta Was this translation helpful? Give feedback.
-
|
If you want the latest version of RabbitMQ to be deployed in your cluster, you should specify the image https://www.rabbitmq.com/kubernetes/operator/using-operator#images |
Beta Was this translation helpful? Give feedback.
-
|
These are the contents of the global:
defaultStorageClass: 'gp3' |
Beta Was this translation helpful? Give feedback.
-
|
@BeyondEvil per our Community Support policy, we will not troubleshoot OAuth 2 for you, in particular together with Bitnami images, Dex and so on. Start with getting We cannot guess how much memory these nodes are allowed to use, what uses it, and so on. You are the first person who claims that using OAuth 2 results in nodes being OOM-killed, and we know for a fact that there are some very large scale installations and even installations that use several IDPs at once. |
Beta Was this translation helpful? Give feedback.
@BeyondEvil per our Community Support policy, we will not troubleshoot OAuth 2 for you, in particular together with Bitnami images, Dex and so on.
Start with getting
4.0.7, then proceed to Reasoning About Memory Use and monitoring and charting the same data.We cannot guess how much memory these nodes are allowed to use, what uses it, and so on. You are the first person who claims that using OAuth 2 results in nodes being OOM-killed, and we know for a fact that there are some very large scale installations and even installations that use several IDPs at once.