Description
What happened?
I have tried to update our grid, from 4.29 to 4.31 at two separate occasions. Both times I see an abnormal high use of CPU in the kubernetes grid. Even when the nodes are idleing.
Going from an idle CPU usage on the nodes of 0.01 cpu eqvivalents, to 100% on 1 cpu per node (they are allowed 3 cpu each if they need, soo it does not max out).
Normal testrun cpu usage on the left in the image. At 07:10 i updated to 4.31 on three nodes.
Here are the CPU usage in table form.. Observe three nodes with 0.99 usage, and 2 with 0.01 (the latter is normal).
I can't see anything in any log that gives any hints on why this is happening.. I tried turning of Tracing (could not see anything in the traces anyway), but that did not do any changes..
I have tried to anonymize the setup files in this bugreport..
Command used to start Selenium Grid with Docker (or Kubernetes)
Example deployment file for one node, using both chrome and edge, and adding a mtlscert:
apiVersion: apps/v1
kind: Deployment
metadata:
name: selenium-magicnode-XXXXYYYY
labels:
app: selenium-magicnode-XXXXYYYY
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 0
selector:
matchLabels:
app: selenium-magicnode-XXXXYYYY
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
labels:
app: selenium-magicnode-XXXXYYYY
spec:
volumes:
- name: dshm
emptyDir:
medium: Memory
sizeLimit: 2Gi
## Video volume
- name: selenium-hub-volume
persistentVolumeClaim:
claimName: selenium-hub-pvc
## Custom certs volume
- name: cert-volume
secret:
secretName: XXXXYYYY
items:
- key: cert.p12
path: cert.p12
- key: password
path: password
containers:
- name: selenium-magicnode-XXXXYYYY
image: localImage
imagePullPolicy: Always
ports:
- containerPort: 5555
- containerPort: 5900
- containerPort: 4444
## Volume mounts
volumeMounts:
- mountPath: /dev/shm
name: dshm
## Video volume mount
- mountPath: "/videos"
name: selenium-hub-volume
## Custom certs volume mount
- name: cert-volume
mountPath: /opt/selenium/certs
readOnly: true
env:
## Custom certs setup
- name: SE_ENABLE_CUSTOM_CERTS
value: "true"
- name: CERT_TRUST_ATTR
value: "TCu,Cu,Tu"
## Event bus setup
- name: SE_EVENT_BUS_HOST
value: "URL"
- name: SE_EVENT_BUS_SUBSCRIBE_PORT
value: "4443"
- name: SE_EVENT_BUS_PUBLISH_PORT
value: "4442"
- name: SE_NODE_GRID_URL
value: "https://URL"
## Stereotype setup
- name: SE_NODE_STEREOTYPE
value: "[{\"browserName\": \"chrome\"}, {\"browserName\": \"MicrosoftEdge\"}]"
- name: SE_NODE_STEREOTYPE_EXTRA
value: "{\"platformName\": \"Linux\", \"character:name\": \"XXXXYYYY\", \"networkname:applicationName\":\"XXXXYYYY\", \"nodename:applicationName\":\"XXXXYYYY\"}"
- name: SE_NODE_NAME
value: "XXXXYYYY"
- name: SE_OTEL_SERVICE_NAME
value: "XXXXYYYY"
- name: SE_NODE_MAX_SESSIONS
value: "1"
- name: SE_NODE_OVERRIDE_MAX_SESSIONS
value: "false"
## General setup
- name: GENERATE_CONFIG
value: "true"
- name: TZ
value: "Europe/Stockholm"
- name: SE_ENABLE_MANAGED_DOWNLOADS
value: "true"
- name: SE_VNC_NO_PASSWORD
value: "1"
## Tracing setup
- name: SE_ENABLE_TRACING
value: "false"
- name: SE_OTEL_TRACES_EXPORTER
value: "otlp"
- name: SE_OTEL_EXPORTER_ENDPOINT
value: "http://selenium-hub-jaeger-svc:4317"
## Video setup
- name: SE_RECORD_VIDEO
value: "true"
- name: SE_VIDEO_FILE_NAME
value: "auto"
## Security context
securityContext:
runAsNonRoot: true
runAsUser: 1200
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
#seccompProfile:
# type: RuntimeDefault
## Probes
readinessProbe:
failureThreshold: 3
httpGet:
path: /status
port: 5555
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
startupProbe:
failureThreshold: 3
httpGet:
path: /status
port: 5555
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
livenessProbe:
failureThreshold: 3
httpGet:
path: /status
port: 5555
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
## Resources (1 cpu per browser at least, and 1 gig of memory per browser)
resources:
limits:
memory: "4Gi"
cpu: "3"
requests:
memory: "100Mi"
cpu: "0.1"
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
imagePullSecrets:
- name: docker-credentials
restartPolicy: Always
schedulerName: default-scheduler
terminationGracePeriodSeconds: 30
Relevant log output
Installing certificates...
2025-04-15T07:08:22.970316001Z Found certificate file
2025-04-15T07:08:22.980665242Z Installing certificate in: /home/seluser/.pki/nssdb
pk12util: no nickname for cert in PKCS12 file.
2025-04-15T07:08:23.015961136Z pk12util: using nickname: XXXXYYYY
2025-04-15T07:08:23.015963188Z pk12util: PKCS12 IMPORT SUCCESSFUL
2025-04-15T07:08:23.021365918Z Virtual environment detected at /opt/venv, activating...
2025-04-15T07:08:23.025128350Z Python 3.12.3
2025-04-15T07:08:23.153845714Z 2025-04-15 09:08:23,153 INFO Included extra file "/etc/supervisor/conf.d/chrome-cleanup.conf" during parsing
2025-04-15T07:08:23.153896896Z 2025-04-15 09:08:23,153 INFO Included extra file "/etc/supervisor/conf.d/edge-cleanup.conf" during parsing
2025-04-15T07:08:23.153899548Z 2025-04-15 09:08:23,153 INFO Included extra file "/etc/supervisor/conf.d/firefox-cleanup.conf" during parsing
2025-04-15T07:08:23.153901289Z 2025-04-15 09:08:23,153 INFO Included extra file "/etc/supervisor/conf.d/recorder.conf" during parsing
2025-04-15 09:08:23,153 INFO Included extra file "/etc/supervisor/conf.d/selenium.conf" during parsing
2025-04-15T07:08:23.153933914Z 2025-04-15 09:08:23,153 INFO Included extra file "/etc/supervisor/conf.d/uploader.conf" during parsing
2025-04-15 09:08:23,156 INFO RPC interface 'supervisor' initialized
2025-04-15T07:08:23.156974433Z 2025-04-15 09:08:23,156 INFO supervisord started with pid 15
2025-04-15T07:08:24.159579444Z 2025-04-15 09:08:24,159 INFO spawned: 'video-ready' with pid 16
2025-04-15T07:08:24.160791824Z 2025-04-15 09:08:24,160 INFO spawned: 'xvfb' with pid 17
2025-04-15T07:08:24.161872245Z 2025-04-15 09:08:24,161 INFO spawned: 'vnc' with pid 18
2025-04-15 09:08:24,162 INFO spawned: 'novnc' with pid 19
2025-04-15T07:08:24.164137266Z 2025-04-15 09:08:24,163 INFO spawned: 'video-recording' with pid 20
2025-04-15T07:08:24.165300122Z 2025-04-15 09:08:24,165 INFO spawned: 'selenium-node' with pid 21
2025-04-15T07:08:24.173657199Z 2025-04-15 09:08:24,173 INFO success: video-ready entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
2025-04-15T07:08:24.173701086Z 2025-04-15 09:08:24,173 INFO success: video-recording entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
2025-04-15T07:08:24.173705054Z 2025-04-15 09:08:24,173 INFO success: selenium-node entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
2025-04-15T07:08:24.184559641Z 2025-04-15 07:08:24,183 [endpoint.checks] - Endpoint http://localhost:5555/status is not available - status code: 000
2025-04-15T07:08:24.186096829Z 2025-04-15 07:08:24,185 [video.recorder] - Video folder exists: /videos
2025-04-15 07:08:24,188 [video.recorder] - Waiting for the display localhost:99.0 is open
2025-04-15T07:08:24.635531632Z Appending Selenium option: --register-period 120
2025-04-15T07:08:24.635557410Z Appending Selenium option: --register-cycle 10
2025-04-15T07:08:24.635562310Z Appending Selenium option: --register-shutdown-on-failure
2025-04-15T07:08:24.635565577Z Appending Selenium option: --heartbeat-period 30
2025-04-15T07:08:24.635608173Z Appending Selenium option: --log-level INFO
2025-04-15T07:08:24.635638529Z Appending Selenium option: --http-logs false
2025-04-15T07:08:24.635709830Z Appending Selenium option: --structured-logs false
2025-04-15T07:08:24.635806723Z Generating Selenium Config
2025-04-15T07:08:24.637985153Z Configuring server...
2025-04-15T07:08:24.637995901Z Setting up SE_NODE_HOST...
Setting up SE_NODE_PORT...
2025-04-15T07:08:24.719084240Z Appending Selenium option: --tracing false
2025-04-15T07:08:24.719124024Z Tracing is disabled
2025-04-15T07:08:24.719135278Z Appending Selenium option: --bind-host false
2025-04-15T07:08:24.719181178Z Appending Selenium option: --config /opt/selenium/config.toml
2025-04-15T07:08:24.719195717Z Selenium Grid Node configuration:
2025-04-15T07:08:24.719952064Z [events]
2025-04-15T07:08:24.719961462Z publish = "tcp://URL:4442"
subscribe = "tcp://URL:4443"
2025-04-15T07:08:24.719965953Z
2025-04-15T07:08:24.719967871Z [node]
2025-04-15T07:08:24.719970125Z grid-url = "https://URL"
2025-04-15T07:08:24.719972171Z session-timeout = 300
2025-04-15T07:08:24.719974326Z override-max-sessions = false
2025-04-15T07:08:24.719976111Z detect-drivers = false
2025-04-15T07:08:24.719978002Z drain-after-session-count = 0
max-sessions = 1
2025-04-15T07:08:24.719981495Z
2025-04-15T07:08:24.719983397Z [[node.driver-configuration]]
2025-04-15T07:08:24.719985216Z display-name = "chrome"
2025-04-15T07:08:24.719988649Z stereotype = '{"browserName": "chrome", "platformName": "Linux", "character:name": "XXXXYYYY", "networkname:applicationName": "XXXXYYYY", "nodename:applicationName": "XXXXYYYY"}'
max-sessions = 1
2025-04-15T07:08:24.720012290Z
2025-04-15T07:08:24.720014083Z [[node.driver-configuration]]
2025-04-15T07:08:24.720016377Z display-name = "MicrosoftEdge"
2025-04-15T07:08:24.720019395Z stereotype = '{"browserName": "MicrosoftEdge", "platformName": "Linux", "character:name": "XXXXYYYY", "networkname:applicationName": "XXXXYYYY", "nodename:applicationName": "XXXXYYYY"}'
2025-04-15T07:08:24.720021222Z max-sessions = 1
2025-04-15T07:08:24.720022874Z
2025-04-15T07:08:24.720197219Z Starting Selenium Grid Node...
2025-04-15T07:08:24.720326612Z Using JAVA_OPTS: -Dwebdriver.remote.enableTracing=false -Dwebdriver.httpclient.version=HTTP_1_1
2025-04-15T07:08:25.197832226Z 2025-04-15 07:08:25,197 [video.recorder] - Display localhost:99.0 is open with dimensions 1920x1080
2025-04-15 09:08:25,197 INFO success: xvfb entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-04-15T07:08:25.198206730Z 2025-04-15 09:08:25,198 INFO success: vnc entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-04-15T07:08:25.198247735Z 2025-04-15 09:08:25,198 INFO success: novnc entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-04-15T07:08:25.199167220Z 2025-04-15 07:08:25,198 [video.recorder] - Waiting for Node endpoint responds
2025-04-15T07:08:25.200223115Z 09:08:25.195 INFO [LoggingOptions.configureLogEncoding] - Using the system default encoding
2025-04-15T07:08:25.204500538Z 09:08:25.204 INFO [LoggingOptions.getTracer] - Using null tracer
2025-04-15T07:08:25.263847842Z 09:08:25.263 INFO [UnboundZmqEventBus.<init>] - Connecting to tcp://url:4442 and tcp://url:4443
2025-04-15T07:08:25.344603556Z 09:08:25.344 INFO [UnboundZmqEventBus.<init>] - Sockets created
09:08:26.346 INFO [UnboundZmqEventBus.<init>] - Event bus ready
09:08:26.492 INFO [NodeServer.createHandlers] - Reporting self as: http://10.57.2.117:5555
09:08:26.495 INFO [LoggingOptions.getTracer] - Using null tracer
09:08:26.508 INFO [NodeOptions.getSessionFactories] - Detected 3 available processors
09:08:26.547 INFO [NodeOptions.report] - Adding MicrosoftEdge for {"browserName": "MicrosoftEdge","character:name": "XXXXYYYY","networkname:applicationName": "XXXXYYYY","nodename:applicationName": "XXXXYYYY","platformName": "linux","se:noVncPort": 7900,"se:vncEnabled": true} 1 times
09:08:26.548 INFO [NodeOptions.report] - Adding chrome for {"browserName": "chrome","character:name": "XXXXYYYY","networkname:applicationName": "XXXXYYYY","nodename:applicationName": "XXXXYYYY","platformName": "linux","se:noVncPort": 7900,"se:vncEnabled": true} 1 times
09:08:26.580 INFO [Node.<init>] - Binding additional locator mechanisms: relative
09:08:26.714 INFO [NodeServer$2.start] - Starting registration process for Node http://10.57.2.117:5555
09:08:26.715 INFO [NodeServer.execute] - Started Selenium node 4.31.0 (revision 4ae8fc9f8a): http://10.57.2.117:5555
09:08:26.736 INFO [NodeServer$2.lambda$start$2] - Sending registration event...
09:08:26.825 INFO [NodeServer.lambda$createHandlers$2] - Node has been added
2025-04-15 07:08:27,218 [video.recorder] - Node endpoint is responding now. Proceeding next steps...
Operating System
Kubernetes / Ranger / Argo
Docker Selenium version (image tag)
4.31