Slow virtual machine list

[notprogrammer]

Hello,

while using the Cockpit I ran into a problem with the list of virtual machines loading slowly. The update didn't help. I read in the discussions that Cockpit has a limit for managing 20 machines. Can you please confirm if this limit actually exists? I couldn't find it directly in the documentation. If so, is it possible to somehow increase or optimize it? He needs to manage up to 60 virtual servers in the cockpit, and we don't want to wait for tens of seconds to load the list of virtual servers.

This problem of virtual server list loading too slowly only occurs when the list is loaded for the first time after login. Every next load is already immediate. I am sending screenshots of what it looks like to log out and log in again on the screenshots attached to this thread as attachments.

BTW: It loads slowly for us even on a server where there are only 19 VMs running + 2 VMs turned off.

journalctl -u cockpit:

writes only cockpit-tls: gnutls_handshake failed: A TLS fatal alert, which I associate with untrusted self-signed
journalctl -u libvirtd:

-- No entries --

The event log says this:

13:56
Failed to open a VPD file '/sys/bus/pci/devices/0000:05:00.1/vpd': Permission denied
virtnodevd
13:56
Failed to open a VPD file '/sys/bus/pci/devices/0000:01:00.1/vpd': Permission denied
virtnodevd
13:56
Failed to open a VPD file '/sys/bus/pci/devices/0000:02:00.1/vpd': Permission denied
virtnodevd
13:56
Failed to open a VPD file '/sys/bus/pci/devices/0000:05:00.0/vpd': Permission denied
virtnodevd
13:56
Failed to open a VPD file '/sys/bus/pci/devices/0000:02:00.0/vpd': Permission denied
virtnodevd
13:56
Failed to open a VPD file '/sys/bus/pci/devices/0000:01:00.0/vpd': Permission denied
virtnodevd
13:56
Failed to open a VPD file '/sys/bus/pci/devices/0000:05:00.1/vpd': Permission denied
virtnodevd
13:56
Failed to open a VPD file '/sys/bus/pci/devices/0000:01:00.1/vpd': Permission denied
virtnodevd
13:56
Failed to open a VPD file '/sys/bus/pci/devices/0000:02:00.1/vpd': Permission denied
virtnodevd
13:56
Failed to open a VPD file '/sys/bus/pci/devices/0000:05:00.0/vpd': Permission denied
virtnodevd
13:56
Failed to open a VPD file '/sys/bus/pci/devices/0000:02:00.0/vpd': Permission denied
virtnodevd
13:56
Failed to open a VPD file '/sys/bus/pci/devices/0000:01:00.0/vpd': Permission denied
virtnodevd
13:56
Failed to open a VPD file '/sys/bus/pci/devices/0000:05:00.1/vpd': Permission denied
virtnodevd

Sample stat:

# stat /sys/bus/pci/devices/0000:05:00.1/vpd
 File: /sys/bus/pci/devices/0000:05:00.1/vpd
 Size: 0 Blocks: 0 IO Block: 4096 regular empty file
Device: 15h/21d Inode: 10166 Links: 1
Access: (0600/-rw-------) Uid: ( 0/ root) Gid: ( 0/ root)
Context: system_u:object_r:sysfs_t:s0
Access: 2024-10-16 14:45:12.595980287 +0200
Modify: 2024-10-16 14:45:12.595980287 +0200
Change: 2024-10-16 14:45:12.595980287 +0200
 Birth:-
# stat /sys/bus/pci/devices/0000:01:00.1/vpd
 File: /sys/bus/pci/devices/0000:01:00.1/vpd
 Size: 0 Blocks: 0 IO Block: 4096 regular empty file
Device: 15h/21d Inode: 9971 Links: 1
Access: (0600/-rw-------) Uid: ( 0/ root) Gid: ( 0/ root)
Context: system_u:object_r:sysfs_t:s0
Access: 2024-10-16 14:45:12.405975751 +0200
Modify: 2024-10-16 14:45:12.405975751 +0200
Change: 2024-10-16 14:45:12.405975751 +0200
 Birth:-
#

Does anyone have experience with multiple VMs in the cockpit please? Also, does the VM list take a long time to load? What could it be and how to solve it?

Thank you for your help.

[mvollmer]

Yes, cockpit-machines feels quite slow when "Loading resources". I'll have a look.

[mvollmer]

Yes, cockpit-machines feels quite slow when "Loading resources". I'll have a look.

Bad news: In my tests, these are the times from logging into Cockpit until the list of VMs is displayed:

   1 VM: 2.7 seconds
101 VMs: 3.5 seconds

So I would argue that 2.7 seconds is slow, but it doesn't get much worse when the number of VMs grows.

Could you share more about what exactly you are doing and when the slowness is there and when it isn't?

This problem of virtual server list loading too slowly only occurs when the list is loaded for the first time after login.

That's a login into Cockpit, right? There is no reboot of the physical server involved or other things, right? What kind of network is there between the web browser that runs Cockpit and the machine that you log into? (Cockpit loads the full XML for each VM, not just the name. If the network is slow enough, this will be noticeable.)

Every next load is already immediate.

What is a "next load"? Do you reload the browser? Or switch between views in the Cockpit application?

[notprogrammer]

The slowness is far away after logging in from my and Kleg's PC at home (we live 150 km apart). I am 100 km from the data center, my colleague is 50 km from the data center. The first loading of the list after login takes time, but every subsequent loading of the list is already instantaneous. But as soon as I log out and log in again, it takes a long time again. "next load" meant switching to another tab in the cockpit and returning to the virtual servers tab.

Yes, that meant logging out and logging in as a user or root user in the cockpit. We restart the server only sometimes after an update, which takes approx. once a year. However, we tried the update to see if it would speed up loading. After the update, we rebooted the server and immediately after the reboot, the list of servers loaded quickly, but that only lasted a while, after a while the list started to be slow again.

I tried removing the surrounding hosts from the cockpit and logging in as root instead of user. And the problem persists. It takes a long time to load the list for the first time after logging in. every other load is now instant until I log out and log in again, when the initial load of the list takes a long time again.

[mvollmer]

Could you run the following script to do some benchmarking? It dumps all the XML of all VMs, which is pretty much Cockpit is doing at start up.

cmd=""
for name in $(virsh list --all --name); do
  cmd="$cmd dumpxml $name; dumpxml --inactive $name; "
done
echo $cmd | virsh

Please run it in the Terminal of the Cockpit instance where you see the slow list loading, so that the same amount of data travels over the same network and also has to go through Cockpit. Doing that in the server that you have setup yesterday just for benchmarking should already be helpful.

I assume your VMs are on the "system" connection of libvirt. Is that true?

With my test setup, 100 shut-off mini VMs, this script completes in about 0.2 seconds. Cockpit itself takes about 1 second to get all its initial data (which is slightly more than just VMs) and also includes XML parsing.

My goal here is to figure out whether the slowness comes from producing and transferring this data, or whether there is something really special in your setup that makes Cockpit very slow.

[notprogrammer]

As I already wrote virsh commands are immediate.

I try this:

#!/bin/bash

# Start time measurement
start_time=$(date +%s)

# Initialize the command variable
cmd=""

# Initialize counters
running_count=0
stopped_count=0

# Iterate over all virtual machines
for name in $(virsh list --all --name); do
  # Check the state of each virtual machine
  state=$(virsh domstate "$name")
  if [[ $state == "running" ]]; then
    ((running_count++)) # Increment running counter
  elif [[ $state == "shut off" ]]; then
    ((stopped_count++)) # Increment stopped counter
  fi

  # Add commands to dump XML configurations
  cmd="$cmd dumpxml $name; dumpxml --inactive $name; "
done

# Execute the commands
echo $cmd | virsh

# End time measurement
end_time=$(date +%s)

# Calculate the total duration
duration=$((end_time - start_time))

# Display the results
echo "Processing took $duration seconds."
echo "Number of running servers: $running_count"
echo "Number of stopped servers: $stopped_count"

Result:

Processing took 1 seconds.
Number of running servers: 18
Number of stopped servers: 3