Wrong resource usage often reported in multiple area

[levindecaro]

ISSUE TYPE

• Bug Report

COMPONENT NAME

Resource Usage

CLOUDSTACK VERSION

4.19

CONFIGURATION

KVM
Advanced Networking
resource.count.running.vms.only=true

OS / ENVIRONMENT

CentOS 7
Oracle Linux 9
KVM

SUMMARY

Multiple user account reports CPU count not released after VM stopped/expunged and inaccurate secondary storage usage reported over time. There are no error been found in management server logs.

STEPS TO REPRODUCE

Spin up vm instance, stop it and delete it.
Create volume snapshot and delete it.

EXPECTED RESULTS

Resource limit usage should be updated and reflect to actual usage, and reconciliation should perform in regular basis. 

ACTUAL RESULTS

Resource count is inaccurate.

[vishesh92]

@levindecaro There is a regular task which reconciles the limits. The period for this task is controlled by resourcecount.check.interval.
There are a few PRs which were merged recently which help reduce the discrepancies and speed up the resource count calculation as well.
#8302
#8903

Over multiple operations there can be some discrepancies. But the regular resource count check should resolve that. Complete solving this requires a complete refactor of the current implementation or adding more locks which can impact the scalability of ACS itself.

[levindecaro]

@vishesh92 I have resourcecount.check.interval=300 inplace, but seems not help for this discrepancies problem.

[rajujith]

@levindecaro have you tried updating resource count for the account after deleting the VM as a workaround?

[levindecaro]

@rajujith i did, but doesn't help

[DaanHoogland]

@levindecaro , several queries (and possible actions)

• is this an intermitted issue or consequent?
• if this is really an issue with usage; for those resources you can check the events in the events table to see if some events were missed. You can enter them manually.
• it sounds though like it is an issue with quota only, is that correct?
• do you see chance to test , in a test environment, with the PRs @vishesh92 refers?

[levindecaro]

@DaanHoogland

So far, I heard it is a intermitted issue from users, when they do some excess create/start/stop/destroy VM within a short period. After hours, the resource never went down, so i need to recalculate their used resource and update db table manually to solve this problem.

For the snapshot usage from secondary storage, there are nothing listed in volume snapshot, but resource limit page shown there are consumed spaces.

I'm no idea how to sort out it is a actual used space or just a orphan record.