CPU to Memory weight based algorithm to order cluster #10997

sudo87 · 2025-06-10T09:27:28Z

Description

This PR introduces new value: "COMBINED" for config: "host.capacityType.to.order.clusters", which will be used to order cluster, host and pods based on CPU and Memory both.

COMBINED will work with "host.capacityType.to.order.clusters.cputomemoryweight" and overall capacity for cluster/pod/host will be computed based on CPU and memory using weight factor.

The allocator will need to first calculate the combined allocation/usage metric (as follows), before sorting the clusters/pods/hosts to return a ordered list of hosts by this metric, for example:
For each host, define metric as:

Metric = CPU * weight + Memory * (1-weight)

Doc PR: apache/cloudstack-documentation#524

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
build/CI
test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

host.capacityType.to.order.clusters config will support new algorithm: COMBINED which will work with host.capacityType.to.order.clusters.cputomemoryweight and capacity will be computed based on CPU and memory both and using weight factor

codecov · 2025-06-10T09:35:36Z

Codecov Report

Attention: Patch coverage is 78.00000% with 33 lines in your changes missing coverage. Please review.

Project coverage is 16.59%. Comparing base (2d669db) to head (99bd592).
Report is 14 commits behind head on main.

Files with missing lines	Patch %	Lines
...rc/main/java/com/cloud/deploy/FirstFitPlanner.java	74.32%	18 Missing and 1 partial ⚠️
...gent/manager/allocator/impl/FirstFitAllocator.java	50.00%	13 Missing ⚠️
.../cloud/configuration/ConfigurationManagerImpl.java	50.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #10997      +/-   ##
============================================
- Coverage     16.60%   16.59%   -0.02%     
- Complexity    13924    13995      +71     
============================================
  Files          5730     5743      +13     
  Lines        508082   510597    +2515     
  Branches      61770    62093     +323     
============================================
+ Hits          84388    84725     +337     
- Misses       414259   416408    +2149     
- Partials       9435     9464      +29

Flag	Coverage Δ
uitests	`3.90% <ø> (-0.04%)`	⬇️
unittests	`17.49% <78.00%> (-0.02%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

github-actions · 2025-06-11T12:37:46Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

sudo87 · 2025-06-12T08:57:54Z

@blueorangutan package

blueorangutan · 2025-06-12T09:00:05Z

@sudo87 a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-06-12T10:15:55Z

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 13745

sudo87 · 2025-06-12T11:58:33Z

@blueorangutan package

blueorangutan · 2025-06-12T12:00:04Z

@sudo87 a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-06-12T13:29:32Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 13751

DaanHoogland · 2025-06-13T14:41:50Z

@sudo87 will you create a doc PR for this as well?

sudo87 · 2025-06-13T15:35:50Z

@sudo87 will you create a doc PR for this as well?

Yes @DaanHoogland, doc pr will be needed for this change.

engine/schema/src/main/java/com/cloud/capacity/dao/CapacityDaoImpl.java

server/src/main/java/com/cloud/agent/manager/allocator/impl/FirstFitAllocator.java

server/src/main/java/com/cloud/deploy/FirstFitPlanner.java

sudo87 · 2025-06-17T16:58:03Z

@blueorangutan package

blueorangutan · 2025-06-17T17:00:04Z

@sudo87 a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-06-17T18:08:51Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 13821

sudo87 · 2025-06-18T03:49:29Z

@blueorangutan test

blueorangutan · 2025-06-18T03:50:03Z

@sudo87 a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2025-06-18T05:31:56Z

[SF] Trillian test result (tid-13554)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 2456 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10997-t13554-kvm-ol8.zip
Smoke tests completed. 8 look OK, 133 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Test File
runTest	`Error`	test_2fa.py
runTest	`Error`	test_account_access.py
runTest	`Error`	test_accounts.py
runTest	`Error`	test_affinity_groups_projects.py
runTest	`Error`	test_affinity_groups.py
runTest	`Error`	test_annotations.py
runTest	`Error`	test_async_job.py
runTest	`Error`	test_attach_multiple_volumes.py
runTest	`Error`	test_backup_recovery_dummy.py
runTest	`Error`	test_backup_recovery_veeam.py
runTest	`Error`	test_direct_download.py
runTest	`Error`	test_certauthority_root.py
runTest	`Error`	test_cluster_drs.py
runTest	`Error`	test_console_endpoint.py
runTest	`Error`	test_create_list_domain_account_project.py
runTest	`Error`	test_create_network.py
runTest	`Error`	test_deploy_vgpu_enabled_vm.py
runTest	`Error`	test_deploy_virtio_scsi_vm.py
runTest	`Error`	test_deploy_vm_extra_config_data.py
runTest	`Error`	test_deploy_vm_iso.py
runTest	`Error`	test_deploy_vm_iso_uefi.py
runTest	`Error`	test_deploy_vm_root_resize.py
runTest	`Error`	test_deploy_vms_in_parallel.py
runTest	`Error`	test_deploy_vms_with_varied_deploymentplanners.py
runTest	`Error`	test_deploy_vm_with_userdata.py
runTest	`Error`	test_diagnostics.py
runTest	`Error`	test_disk_offerings.py
runTest	`Error`	test_disk_provisioning_types.py
runTest	`Error`	test_domain_disk_offerings.py
runTest	`Error`	test_domain_network_offerings.py
runTest	`Error`	test_domain_service_offerings.py
runTest	`Error`	test_guest_os.py
runTest	`Error`	test_domain_vpc_offerings.py
runTest	`Error`	test_enable_account_settings_for_domain.py
runTest	`Error`	test_metrics_api.py
runTest	`Error`	test_events_resource.py
runTest	`Error`	test_gateway_on_shared_networks.py
runTest	`Error`	test_global_acls.py
runTest	`Error`	test_global_settings.py
runTest	`Error`	test_guest_vlan_range.py
runTest	`Error`	test_host_control_state.py
runTest	`Error`	test_hostha_simulator.py
runTest	`Error`	test_host_ping.py
runTest	`Error`	test_image_store_object_migration.py
runTest	`Error`	test_import_unmanage_volumes.py
runTest	`Error`	test_internal_lb.py
runTest	`Error`	test_ipv4_routing.py
runTest	`Error`	test_ipv6_infra.py
runTest	`Error`	test_iso.py
runTest	`Error`	test_kubernetes_clusters.py
runTest	`Error`	test_kubernetes_supported_versions.py
runTest	`Error`	test_list_accounts.py
runTest	`Error`	test_list_disk_offerings.py
runTest	`Error`	test_list_domains.py
runTest	`Error`	test_list_hosts.py
runTest	`Error`	test_list_ids_parameter.py
runTest	`Error`	test_list_service_offerings.py
runTest	`Error`	test_list_storage_pools.py
runTest	`Error`	test_list_volumes.py
runTest	`Error`	test_loadbalance.py
runTest	`Error`	test_login.py
runTest	`Error`	test_migration.py
runTest	`Error`	test_ms_maintenance_and_safe_shutdown.py
runTest	`Error`	test_multipleips_per_nic.py
runTest	`Error`	test_nested_virtualization.py
runTest	`Error`	test_network_acl.py
runTest	`Error`	test_network_ipv6.py
runTest	`Error`	test_network_permissions.py
runTest	`Error`	test_network.py
runTest	`Error`	test_nic_adapter_type.py
runTest	`Error`	test_nic.py
runTest	`Error`	test_non_contigiousvlan.py
runTest	`Error`	test_nonstrict_affinity_group.py
runTest	`Error`	test_outofbandmanagement_nestedplugin.py
runTest	`Error`	test_outofbandmanagement.py
runTest	`Error`	test_over_provisioning.py
runTest	`Error`	test_password_server.py
runTest	`Error`	test_persistent_network.py
runTest	`Error`	test_portable_publicip.py
runTest	`Error`	test_portforwardingrules.py
runTest	`Error`	test_primary_storage.py
runTest	`Error`	test_primary_storage_scope.py
runTest	`Error`	test_privategw_acl_ovs_gre.py
runTest	`Error`	test_privategw_acl.py
runTest	`Error`	test_projects.py
runTest	`Error`	test_public_ip_range.py
runTest	`Error`	test_purge_expunged_vms.py
runTest	`Error`	test_pvlan.py
runTest	`Error`	test_quarantined_ips.py
runTest	`Error`	test_regions.py
runTest	`Error`	test_register_userdata.py
runTest	`Error`	test_reset_configuration_settings.py
runTest	`Error`	test_reset_vm_on_reboot.py
runTest	`Error`	test_resource_accounting.py
runTest	`Error`	test_resource_detail.py
runTest	`Error`	test_resource_names.py
runTest	`Error`	test_restore_vm.py
runTest	`Error`	test_router_dhcphosts.py
runTest	`Error`	test_router_dns.py
runTest	`Error`	test_router_dnsservice.py
runTest	`Error`	test_routers_iptables_default_policy.py
runTest	`Error`	test_routers_network_ops.py
runTest	`Error`	test_routers.py
runTest	`Error`	test_scale_vm.py
runTest	`Error`	test_secondary_storage.py
runTest	`Error`	test_service_offerings.py
runTest	`Error`	test_set_sourcenat.py
runTest	`Error`	test_sharedfs_lifecycle.py
runTest	`Error`	test_snapshots.py
runTest	`Error`	test_ssvm.py
runTest	`Error`	test_storage_policy.py
runTest	`Error`	test_templates.py
runTest	`Error`	test_update_security_group.py
runTest	`Error`	test_usage_events.py
runTest	`Error`	test_usage.py
runTest	`Error`	test_vm_autoscaling.py
runTest	`Error`	test_vm_deployment_planner.py
runTest	`Error`	test_vm_life_cycle.py
runTest	`Error`	test_vm_lifecycle_unmanage_import.py
runTest	`Error`	test_vm_schedule.py
runTest	`Error`	test_vm_snapshot_kvm.py
runTest	`Error`	test_vm_snapshots.py
runTest	`Error`	test_vm_strict_host_tags.py
runTest	`Error`	test_vnf_templates.py
runTest	`Error`	test_volumes.py
runTest	`Error`	test_vpc_ipv6.py
runTest	`Error`	test_vpc_redundant.py
runTest	`Error`	test_vpc_router_nics.py
runTest	`Error`	test_vpc_vpn.py
runTest	`Error`	test_webhook_delivery.py
runTest	`Error`	test_webhook_lifecycle.py
runTest	`Error`	test_host_maintenance.py
runTest	`Error`	test_hostha_kvm.py

weizhouapache

@blueorangutan package

engine/components-api/src/main/java/com/cloud/configuration/ConfigurationManager.java

sudo87 · 2025-06-19T05:01:10Z

@blueorangutan package

blueorangutan · 2025-06-19T05:02:05Z

@sudo87 a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-06-19T06:26:02Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 13835

weizhouapache

code lgtm

not tested yet

sudo87 · 2025-06-19T09:53:30Z

@blueorangutan package

blueorangutan · 2025-06-19T09:54:04Z

@sudo87 a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-06-19T11:27:45Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 13843

boring-cyborg bot added the component:api label Jun 10, 2025

minor changes

88c0804

github-actions bot added the status:has-conflicts label Jun 11, 2025

resolve merge conflict

4b0b57f

github-actions bot removed the status:has-conflicts label Jun 11, 2025

sudo87 added 2 commits June 11, 2025 18:42

Merge branch 'main' into clusterOrderVMAlloc

815c0e4

Merge branch 'main' into clusterOrderVMAlloc

f293806

sudo87 marked this pull request as ready for review June 13, 2025 04:30

sudo87 requested a review from weizhouapache June 13, 2025 04:30

weizhouapache reviewed Jun 16, 2025

View reviewed changes

add unit tests

c8b6734

weizhouapache reviewed Jun 18, 2025

View reviewed changes

engine/components-api/src/main/java/com/cloud/configuration/ConfigurationManager.java Outdated Show resolved Hide resolved

weizhouapache approved these changes Jun 19, 2025

View reviewed changes

update desc and add validation

99bd592

sudo87 mentioned this pull request Jun 20, 2025

Add cluster ordering by host capacityType doc apache/cloudstack-documentation#524

Open

CPU to Memory weight based algorithm to order cluster #10997

Are you sure you want to change the base?

CPU to Memory weight based algorithm to order cluster #10997

Uh oh!

Conversation

sudo87 commented Jun 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

codecov bot commented Jun 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions bot commented Jun 11, 2025

Uh oh!

sudo87 commented Jun 12, 2025

Uh oh!

blueorangutan commented Jun 12, 2025

Uh oh!

blueorangutan commented Jun 12, 2025

Uh oh!

sudo87 commented Jun 12, 2025

Uh oh!

blueorangutan commented Jun 12, 2025

Uh oh!

blueorangutan commented Jun 12, 2025

Uh oh!

DaanHoogland commented Jun 13, 2025

Uh oh!

sudo87 commented Jun 13, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sudo87 commented Jun 17, 2025

Uh oh!

blueorangutan commented Jun 17, 2025

Uh oh!

blueorangutan commented Jun 17, 2025

Uh oh!

sudo87 commented Jun 18, 2025

Uh oh!

blueorangutan commented Jun 18, 2025

Uh oh!

blueorangutan commented Jun 18, 2025

Uh oh!

weizhouapache left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sudo87 commented Jun 19, 2025

Uh oh!

blueorangutan commented Jun 19, 2025

Uh oh!

blueorangutan commented Jun 19, 2025

Uh oh!

weizhouapache left a comment

Choose a reason for hiding this comment

Uh oh!

sudo87 commented Jun 19, 2025

Uh oh!

blueorangutan commented Jun 19, 2025

Uh oh!

blueorangutan commented Jun 19, 2025

Uh oh!

Uh oh!

sudo87 commented Jun 10, 2025 •

edited

Loading

codecov bot commented Jun 10, 2025 •

edited

Loading