Skip to content

[🐛 Bug]: High CPU usage with 4.31 compared to 4.30 #2788

Open
@parholmdahl

Description

@parholmdahl

What happened?

I have tried to update our grid, from 4.29 to 4.31 at two separate occasions. Both times I see an abnormal high use of CPU in the kubernetes grid. Even when the nodes are idleing.

Going from an idle CPU usage on the nodes of 0.01 cpu eqvivalents, to 100% on 1 cpu per node (they are allowed 3 cpu each if they need, soo it does not max out).

Image

Normal testrun cpu usage on the left in the image. At 07:10 i updated to 4.31 on three nodes.

Image

Here are the CPU usage in table form.. Observe three nodes with 0.99 usage, and 2 with 0.01 (the latter is normal).

I can't see anything in any log that gives any hints on why this is happening.. I tried turning of Tracing (could not see anything in the traces anyway), but that did not do any changes..

I have tried to anonymize the setup files in this bugreport..

Command used to start Selenium Grid with Docker (or Kubernetes)

Example deployment file for one node, using both chrome and edge, and adding a mtlscert: 

apiVersion: apps/v1
kind: Deployment
metadata:
  name: selenium-magicnode-XXXXYYYY
  labels:
    app: selenium-magicnode-XXXXYYYY
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 0
  selector:
    matchLabels:
      app: selenium-magicnode-XXXXYYYY
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate      
  template:
    metadata:
      labels:
        app: selenium-magicnode-XXXXYYYY
    spec:


      volumes:
      - name: dshm
        emptyDir:
          medium: Memory
          sizeLimit: 2Gi

      ## Video volume
      - name: selenium-hub-volume
        persistentVolumeClaim:
          claimName: selenium-hub-pvc
      
      ## Custom certs volume
      - name: cert-volume
        secret:
          secretName: XXXXYYYY
          items:
          - key: cert.p12
            path: cert.p12
          - key: password
            path: password                

      containers:
      - name: selenium-magicnode-XXXXYYYY
        image: localImage
        imagePullPolicy: Always
        ports:
          - containerPort: 5555
          - containerPort: 5900
          - containerPort: 4444



        ## Volume mounts
        volumeMounts:
          - mountPath: /dev/shm
            name: dshm
          ## Video volume mount
          - mountPath: "/videos"
            name: selenium-hub-volume
          ## Custom certs volume mount
          - name: cert-volume
            mountPath: /opt/selenium/certs
            readOnly: true       


        env:
          ## Custom certs setup
          - name: SE_ENABLE_CUSTOM_CERTS
            value: "true"
          - name: CERT_TRUST_ATTR
            value: "TCu,Cu,Tu"       


          ## Event bus setup
          - name: SE_EVENT_BUS_HOST
            value: "URL"
          - name: SE_EVENT_BUS_SUBSCRIBE_PORT
            value: "4443"
          - name: SE_EVENT_BUS_PUBLISH_PORT
            value: "4442"
          - name: SE_NODE_GRID_URL
            value: "https://URL"


          ## Stereotype setup
          - name: SE_NODE_STEREOTYPE
            value: "[{\"browserName\": \"chrome\"}, {\"browserName\": \"MicrosoftEdge\"}]"
          - name: SE_NODE_STEREOTYPE_EXTRA
            value: "{\"platformName\": \"Linux\", \"character:name\": \"XXXXYYYY\", \"networkname:applicationName\":\"XXXXYYYY\", \"nodename:applicationName\":\"XXXXYYYY\"}"
          - name: SE_NODE_NAME
            value: "XXXXYYYY"
          - name: SE_OTEL_SERVICE_NAME
            value: "XXXXYYYY"       
          - name: SE_NODE_MAX_SESSIONS
            value: "1"
          - name: SE_NODE_OVERRIDE_MAX_SESSIONS
            value: "false"


          ## General setup
          - name: GENERATE_CONFIG
            value: "true"
          - name: TZ
            value: "Europe/Stockholm"
          - name: SE_ENABLE_MANAGED_DOWNLOADS
            value: "true"
          - name: SE_VNC_NO_PASSWORD
            value: "1"


          ## Tracing setup
          - name: SE_ENABLE_TRACING
            value: "false"
          - name: SE_OTEL_TRACES_EXPORTER
            value: "otlp"
          - name: SE_OTEL_EXPORTER_ENDPOINT
            value: "http://selenium-hub-jaeger-svc:4317"


          ## Video setup
          - name: SE_RECORD_VIDEO
            value: "true"
          - name: SE_VIDEO_FILE_NAME
            value: "auto"


        ## Security context
        securityContext:
          runAsNonRoot: true
          runAsUser: 1200
          allowPrivilegeEscalation: false
          capabilities:
            drop: 
              - ALL  
          #seccompProfile: 
          #  type: RuntimeDefault


        ## Probes
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /status
            port: 5555
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        startupProbe:
          failureThreshold: 3
          httpGet:
            path: /status
            port: 5555
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /status
            port: 5555
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1

        ## Resources (1 cpu per browser at least, and 1 gig of memory per browser)
        resources:
          limits:
            memory: "4Gi"
            cpu: "3"
          requests:
            memory: "100Mi"
            cpu: "0.1"


        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      imagePullSecrets:
      - name: docker-credentials
      restartPolicy: Always
      schedulerName: default-scheduler
      terminationGracePeriodSeconds: 30

Relevant log output

Installing certificates...
2025-04-15T07:08:22.970316001Z Found certificate file
2025-04-15T07:08:22.980665242Z Installing certificate in: /home/seluser/.pki/nssdb
pk12util: no nickname for cert in PKCS12 file.
2025-04-15T07:08:23.015961136Z pk12util: using nickname: XXXXYYYY
2025-04-15T07:08:23.015963188Z pk12util: PKCS12 IMPORT SUCCESSFUL
2025-04-15T07:08:23.021365918Z Virtual environment detected at /opt/venv, activating...
2025-04-15T07:08:23.025128350Z Python 3.12.3
2025-04-15T07:08:23.153845714Z 2025-04-15 09:08:23,153 INFO Included extra file "/etc/supervisor/conf.d/chrome-cleanup.conf" during parsing
2025-04-15T07:08:23.153896896Z 2025-04-15 09:08:23,153 INFO Included extra file "/etc/supervisor/conf.d/edge-cleanup.conf" during parsing
2025-04-15T07:08:23.153899548Z 2025-04-15 09:08:23,153 INFO Included extra file "/etc/supervisor/conf.d/firefox-cleanup.conf" during parsing
2025-04-15T07:08:23.153901289Z 2025-04-15 09:08:23,153 INFO Included extra file "/etc/supervisor/conf.d/recorder.conf" during parsing
2025-04-15 09:08:23,153 INFO Included extra file "/etc/supervisor/conf.d/selenium.conf" during parsing
2025-04-15T07:08:23.153933914Z 2025-04-15 09:08:23,153 INFO Included extra file "/etc/supervisor/conf.d/uploader.conf" during parsing
2025-04-15 09:08:23,156 INFO RPC interface 'supervisor' initialized
2025-04-15T07:08:23.156974433Z 2025-04-15 09:08:23,156 INFO supervisord started with pid 15
2025-04-15T07:08:24.159579444Z 2025-04-15 09:08:24,159 INFO spawned: 'video-ready' with pid 16
2025-04-15T07:08:24.160791824Z 2025-04-15 09:08:24,160 INFO spawned: 'xvfb' with pid 17
2025-04-15T07:08:24.161872245Z 2025-04-15 09:08:24,161 INFO spawned: 'vnc' with pid 18
2025-04-15 09:08:24,162 INFO spawned: 'novnc' with pid 19
2025-04-15T07:08:24.164137266Z 2025-04-15 09:08:24,163 INFO spawned: 'video-recording' with pid 20
2025-04-15T07:08:24.165300122Z 2025-04-15 09:08:24,165 INFO spawned: 'selenium-node' with pid 21
2025-04-15T07:08:24.173657199Z 2025-04-15 09:08:24,173 INFO success: video-ready entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
2025-04-15T07:08:24.173701086Z 2025-04-15 09:08:24,173 INFO success: video-recording entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
2025-04-15T07:08:24.173705054Z 2025-04-15 09:08:24,173 INFO success: selenium-node entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
2025-04-15T07:08:24.184559641Z 2025-04-15 07:08:24,183 [endpoint.checks] - Endpoint http://localhost:5555/status is not available - status code: 000
2025-04-15T07:08:24.186096829Z 2025-04-15 07:08:24,185 [video.recorder] - Video folder exists: /videos
2025-04-15 07:08:24,188 [video.recorder] - Waiting for the display localhost:99.0 is open
2025-04-15T07:08:24.635531632Z Appending Selenium option: --register-period 120
2025-04-15T07:08:24.635557410Z Appending Selenium option: --register-cycle 10
2025-04-15T07:08:24.635562310Z Appending Selenium option: --register-shutdown-on-failure 
2025-04-15T07:08:24.635565577Z Appending Selenium option: --heartbeat-period 30
2025-04-15T07:08:24.635608173Z Appending Selenium option: --log-level INFO
2025-04-15T07:08:24.635638529Z Appending Selenium option: --http-logs false
2025-04-15T07:08:24.635709830Z Appending Selenium option: --structured-logs false
2025-04-15T07:08:24.635806723Z Generating Selenium Config
2025-04-15T07:08:24.637985153Z Configuring server...
2025-04-15T07:08:24.637995901Z Setting up SE_NODE_HOST...
Setting up SE_NODE_PORT...
2025-04-15T07:08:24.719084240Z Appending Selenium option: --tracing false
2025-04-15T07:08:24.719124024Z Tracing is disabled
2025-04-15T07:08:24.719135278Z Appending Selenium option: --bind-host false
2025-04-15T07:08:24.719181178Z Appending Selenium option: --config /opt/selenium/config.toml
2025-04-15T07:08:24.719195717Z Selenium Grid Node configuration: 
2025-04-15T07:08:24.719952064Z [events]
2025-04-15T07:08:24.719961462Z     publish = "tcp://URL:4442"
    subscribe = "tcp://URL:4443"
2025-04-15T07:08:24.719965953Z     
2025-04-15T07:08:24.719967871Z [node]
2025-04-15T07:08:24.719970125Z grid-url = "https://URL"
2025-04-15T07:08:24.719972171Z session-timeout = 300
2025-04-15T07:08:24.719974326Z override-max-sessions = false
2025-04-15T07:08:24.719976111Z detect-drivers = false
2025-04-15T07:08:24.719978002Z drain-after-session-count = 0
max-sessions = 1
2025-04-15T07:08:24.719981495Z 
2025-04-15T07:08:24.719983397Z [[node.driver-configuration]]
2025-04-15T07:08:24.719985216Z display-name = "chrome"
2025-04-15T07:08:24.719988649Z stereotype = '{"browserName": "chrome", "platformName": "Linux", "character:name": "XXXXYYYY", "networkname:applicationName": "XXXXYYYY", "nodename:applicationName": "XXXXYYYY"}'
max-sessions = 1
2025-04-15T07:08:24.720012290Z 
2025-04-15T07:08:24.720014083Z [[node.driver-configuration]]
2025-04-15T07:08:24.720016377Z display-name = "MicrosoftEdge"
2025-04-15T07:08:24.720019395Z stereotype = '{"browserName": "MicrosoftEdge", "platformName": "Linux", "character:name": "XXXXYYYY", "networkname:applicationName": "XXXXYYYY", "nodename:applicationName": "XXXXYYYY"}'
2025-04-15T07:08:24.720021222Z max-sessions = 1
2025-04-15T07:08:24.720022874Z 
2025-04-15T07:08:24.720197219Z Starting Selenium Grid Node...
2025-04-15T07:08:24.720326612Z Using JAVA_OPTS:  -Dwebdriver.remote.enableTracing=false -Dwebdriver.httpclient.version=HTTP_1_1
2025-04-15T07:08:25.197832226Z 2025-04-15 07:08:25,197 [video.recorder] - Display localhost:99.0 is open with dimensions 1920x1080
2025-04-15 09:08:25,197 INFO success: xvfb entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-04-15T07:08:25.198206730Z 2025-04-15 09:08:25,198 INFO success: vnc entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-04-15T07:08:25.198247735Z 2025-04-15 09:08:25,198 INFO success: novnc entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-04-15T07:08:25.199167220Z 2025-04-15 07:08:25,198 [video.recorder] - Waiting for Node endpoint responds
2025-04-15T07:08:25.200223115Z 09:08:25.195 INFO [LoggingOptions.configureLogEncoding] - Using the system default encoding
2025-04-15T07:08:25.204500538Z 09:08:25.204 INFO [LoggingOptions.getTracer] - Using null tracer
2025-04-15T07:08:25.263847842Z 09:08:25.263 INFO [UnboundZmqEventBus.<init>] - Connecting to tcp://url:4442 and tcp://url:4443
2025-04-15T07:08:25.344603556Z 09:08:25.344 INFO [UnboundZmqEventBus.<init>] - Sockets created
09:08:26.346 INFO [UnboundZmqEventBus.<init>] - Event bus ready
09:08:26.492 INFO [NodeServer.createHandlers] - Reporting self as: http://10.57.2.117:5555
09:08:26.495 INFO [LoggingOptions.getTracer] - Using null tracer
09:08:26.508 INFO [NodeOptions.getSessionFactories] - Detected 3 available processors
09:08:26.547 INFO [NodeOptions.report] - Adding MicrosoftEdge for {"browserName": "MicrosoftEdge","character:name": "XXXXYYYY","networkname:applicationName": "XXXXYYYY","nodename:applicationName": "XXXXYYYY","platformName": "linux","se:noVncPort": 7900,"se:vncEnabled": true} 1 times
09:08:26.548 INFO [NodeOptions.report] - Adding chrome for {"browserName": "chrome","character:name": "XXXXYYYY","networkname:applicationName": "XXXXYYYY","nodename:applicationName": "XXXXYYYY","platformName": "linux","se:noVncPort": 7900,"se:vncEnabled": true} 1 times
09:08:26.580 INFO [Node.<init>] - Binding additional locator mechanisms: relative
09:08:26.714 INFO [NodeServer$2.start] - Starting registration process for Node http://10.57.2.117:5555
09:08:26.715 INFO [NodeServer.execute] - Started Selenium node 4.31.0 (revision 4ae8fc9f8a): http://10.57.2.117:5555
09:08:26.736 INFO [NodeServer$2.lambda$start$2] - Sending registration event...
09:08:26.825 INFO [NodeServer.lambda$createHandlers$2] - Node has been added
2025-04-15 07:08:27,218 [video.recorder] - Node endpoint is responding now. Proceeding next steps...

Operating System

Kubernetes / Ranger / Argo

Docker Selenium version (image tag)

4.31

Selenium Grid chart version (chart version)

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions