Storage domain creation by passing a multiple Ceph node

[kushagra-e02562]

When we are creating a new storage domain of type POSIX Compliant FS with passing multiple ceph node in the path(since all these nodes are part of the cluster), for eg. path: [abcd:abcd:abcd::23],[abcd:abcd:abcd::22],[abcd:abcd:abcd::21]:6789:/volumes/kush/threetest/cb8a69c4-9c9a-4a2d-9a4a-7b9a5ff14c74, we are able t o create the storage domain. It is coming in active state.
When we create a disk using this storage domain, the disk is not coming in active state. Instead it is in illegal state.

On parsing the logs in the file /var/log/ovirt-engine/engine.log on the hosted engine node, following is the error:

2023-01-09 12:40:13,815+05 INFO  [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] (EE-ManagedScheduledExecutorService-engineThreadMonitoringThreadPool-Thread-1) [] Thread pool 'default' is using 0 threads out of 1, 5 threads waiting for tasks.
2023-01-09 12:40:13,815+05 INFO  [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] (EE-ManagedScheduledExecutorService-engineThreadMonitoringThreadPool-Thread-1) [] Thread pool 'engine' is using 0 threads out of 500, 10 threads waiting for tasks and 0 tasks in queue.
2023-01-09 12:40:13,815+05 INFO  [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] (EE-ManagedScheduledExecutorService-engineThreadMonitoringThreadPool-Thread-1) [] Thread pool 'engineScheduledThreadPool' is using 0 threads out of 1, 100 threads waiting for tasks.
2023-01-09 12:40:13,815+05 INFO  [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] (EE-ManagedScheduledExecutorService-engineThreadMonitoringThreadPool-Thread-1) [] Thread pool 'engineThreadMonitoringThreadPool' is using 1 threads out of 1, 0 threads waiting for tasks.
2023-01-09 12:40:13,815+05 INFO  [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] (EE-ManagedScheduledExecutorService-engineThreadMonitoringThreadPool-Thread-1) [] Thread pool 'hostUpdatesChecker' is using 0 threads out of 5, 4 threads waiting for tasks.
2023-01-09 12:40:17,706+05 INFO  [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-4) [e4ee0883-61f7-4940-9545-b2f36734df04] Lock Acquired to object 'EngineLock:{exclusiveLocks='[]', sharedLocks='[]'}'
2023-01-09 12:40:17,742+05 INFO  [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-4) [e4ee0883-61f7-4940-9545-b2f36734df04] Running command: TransferDiskImageCommand internal: false. Entities affected :  ID: b0d10f22-f913-4c05-8d44-084f775ae87d Type: StorageAction group CREATE_DISK with role type USER
2023-01-09 12:40:17,742+05 INFO  [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-4) [e4ee0883-61f7-4940-9545-b2f36734df04] Creating ImageTransfer entity for command 'df2c9d30-e452-492d-b07c-b4b1232c9de0', proxyEnabled: true
2023-01-09 12:40:17,783+05 INFO  [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-4) [e4ee0883-61f7-4940-9545-b2f36734df04] Creating disk image
2023-01-09 12:40:17,909+05 INFO  [org.ovirt.engine.core.bll.storage.disk.AddDiskCommand] (default task-4) [e4ee0883-61f7-4940-9545-b2f36734df04] Running command: AddDiskCommand internal: true. Entities affected :  ID: b0d10f22-f913-4c05-8d44-084f775ae87d Type: StorageAction group CREATE_DISK with role type USER
2023-01-09 12:40:18,034+05 INFO  [org.ovirt.engine.core.bll.storage.disk.image.AddImageFromScratchCommand] (default task-4) [e4ee0883-61f7-4940-9545-b2f36734df04] Running command: AddImageFromScratchCommand internal: true. Entities affected :  ID: b0d10f22-f913-4c05-8d44-084f775ae87d Type: Storage
2023-01-09 12:40:18,201+05 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.CreateVolumeVDSCommand] (default task-4) [e4ee0883-61f7-4940-9545-b2f36734df04] START, CreateVolumeVDSCommand( CreateVolumeVDSCommandParameters:{storagePoolId='537e6452-8b7a-11ed-a21c-00163e750630', ignoreFailoverLimit='false', storageDomainId='b0d10f22-f913-4c05-8d44-084f775ae87d', imageGroupId='6f649bd6-2235-4ea8-822b-3b47308b3425', imageSizeInBytes='117440512', volumeFormat='COW', newImageId='032c711e-c488-4f2d-ae7c-363db5b6ffba', imageType='Sparse', newImageDescription='{"DiskAlias":"cirrorstrialpossix2","DiskDescription":"this is a trial for possix"}', imageInitialSizeInBytes='0', imageId='00000000-0000-0000-0000-000000000000', sourceImageGroupId='00000000-0000-0000-0000-000000000000', shouldAddBitmaps='false'}), log id: 6f257c25
2023-01-09 12:40:18,840+05 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.CreateVolumeVDSCommand] (default task-4) [e4ee0883-61f7-4940-9545-b2f36734df04] FINISH, CreateVolumeVDSCommand, return: 032c711e-c488-4f2d-ae7c-363db5b6ffba, log id: 6f257c25
2023-01-09 12:40:18,842+05 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (default task-4) [e4ee0883-61f7-4940-9545-b2f36734df04] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command 'b0b78998-d71b-4dec-970d-0fbfa43d2e19'
2023-01-09 12:40:18,842+05 INFO  [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (default task-4) [e4ee0883-61f7-4940-9545-b2f36734df04] CommandMultiAsyncTasks::attachTask: Attaching task '660f08b4-c380-4ece-86ee-41db40e6bb22' to command 'b0b78998-d71b-4dec-970d-0fbfa43d2e19'.
2023-01-09 12:40:19,034+05 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (default task-4) [e4ee0883-61f7-4940-9545-b2f36734df04] Adding task '660f08b4-c380-4ece-86ee-41db40e6bb22' (Parent Command 'AddImageFromScratch', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters'), polling hasn't started yet..
2023-01-09 12:40:19,127+05 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (default task-4) [e4ee0883-61f7-4940-9545-b2f36734df04] BaseAsyncTask::startPollingTask: Starting to poll task '660f08b4-c380-4ece-86ee-41db40e6bb22'.
2023-01-09 12:40:19,385+05 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-4) [e4ee0883-61f7-4940-9545-b2f36734df04] EVENT_ID: ADD_DISK_INTERNAL(2,036), Add-Disk operation of 'cirrorstrialpossix2' was initiated by the system.
2023-01-09 12:40:19,660+05 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-4) [e4ee0883-61f7-4940-9545-b2f36734df04] EVENT_ID: TRANSFER_IMAGE_INITIATED(1,031), Image Upload with disk cirrorstrialpossix2 was initiated by admin@internal-authz.
2023-01-09 12:40:20,378+05 INFO  [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-24) [e4ee0883-61f7-4940-9545-b2f36734df04] Waiting for disk to be added for image transfer command 'df2c9d30-e452-492d-b07c-b4b1232c9de0'
2023-01-09 12:40:20,378+05 INFO  [org.ovirt.engine.core.bll.storage.disk.AddDiskCommandCallback] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-24) [e4ee0883-61f7-4940-9545-b2f36734df04] Command 'AddDisk' (id: '674288b3-6da2-4458-9113-f64afc1e6685') waiting on child command id: 'b0b78998-d71b-4dec-970d-0fbfa43d2e19' type:'AddImageFromScratch' to complete
2023-01-09 12:40:20,497+05 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-50) [] Setting new tasks map. The map contains now 1 tasks
2023-01-09 12:40:23,737+05 INFO  [org.ovirt.engine.core.bll.storage.disk.image.TransferImageStatusCommand] (default task-4) [cd62aea8-2e37-4af1-9b0b-21a740436ca3] Running command: TransferImageStatusCommand internal: false. Entities affected :  ID: b0d10f22-f913-4c05-8d44-084f775ae87d Type: SystemAction group CREATE_DISK with role type USER
2023-01-09 12:40:24,388+05 INFO  [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-85) [e4ee0883-61f7-4940-9545-b2f36734df04] Waiting for disk to be added for image transfer command 'df2c9d30-e452-492d-b07c-b4b1232c9de0'
2023-01-09 12:40:24,389+05 INFO  [org.ovirt.engine.core.bll.storage.disk.AddDiskCommandCallback] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-85) [e4ee0883-61f7-4940-9545-b2f36734df04] Command 'AddDisk' (id: '674288b3-6da2-4458-9113-f64afc1e6685') waiting on child command id: 'b0b78998-d71b-4dec-970d-0fbfa43d2e19' type:'AddImageFromScratch' to complete
2023-01-09 12:40:25,757+05 INFO  [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-89) [] Polling and updating Async Tasks: 1 tasks, 1 tasks to poll now
2023-01-09 12:40:25,917+05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-89) [] Failed in 'HSMGetAllTasksStatusesVDS' method
2023-01-09 12:40:25,948+05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-89) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM ovirt-host command HSMGetAllTasksStatusesVDS failed: value=ParamsList: sep , in /rhev/data-center/mnt/[abcd:abcd:abcd::23],[abcd:abcd:abcd::22],[abcd:abcd:abcd::21]:6789:_volumes_kush_threetest_cb8a69c4-9c9a-4a2d-9a4a-7b9a5ff14c74/b0d10f22-f913-4c05-8d44-084f775ae87d/images/6f649bd6-2235-4ea8-822b-3b47308b3425 abortedcode=100
2023-01-09 12:40:25,948+05 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-89) [] SPMAsyncTask::PollTask: Polling task '660f08b4-c380-4ece-86ee-41db40e6bb22' (Parent Command 'AddImageFromScratch', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') returned status 'finished', result 'cleanSuccess'.
2023-01-09 12:40:25,948+05 ERROR [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-89) [] BaseAsyncTask::logEndTaskFailure: Task '660f08b4-c380-4ece-86ee-41db40e6bb22' (Parent Command 'AddImageFromScratch', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') ended with failure:
-- Result: 'cleanSuccess'
-- Message: 'VDSGenericException: VDSErrorException: Failed to HSMGetAllTasksStatusesVDS, error = value=ParamsList: sep , in /rhev/data-center/mnt/[abcd:abcd:abcd::23],[abcd:abcd:abcd::22],[abcd:abcd:abcd::21]:6789:_volumes_kush_threetest_cb8a69c4-9c9a-4a2d-9a4a-7b9a5ff14c74/b0d10f22-f913-4c05-8d44-084f775ae87d/images/6f649bd6-2235-4ea8-822b-3b47308b3425 abortedcode=100, code = 100',
-- Exception: 'VDSGenericException: VDSErrorException: Failed to HSMGetAllTasksStatusesVDS, error = value=ParamsList: sep , in /rhev/data-center/mnt/[abcd:abcd:abcd::23],[abcd:abcd:abcd::22],[abcd:abcd:abcd::21]:6789:_volumes_kush_threetest_cb8a69c4-9c9a-4a2d-9a4a-7b9a5ff14c74/b0d10f22-f913-4c05-8d44-084f775ae87d/images/6f649bd6-2235-4ea8-822b-3b47308b3425 abortedcode=100, code = 100'
2023-01-09 12:40:25,949+05 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-89) [] CommandAsyncTask::endActionIfNecessary: All tasks of command 'b0b78998-d71b-4dec-970d-0fbfa43d2e19' has ended -> executing 'endAction'
2023-01-09 12:40:25,949+05 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-89) [] CommandAsyncTask::endAction: Ending action for '1' tasks (command ID: 'b0b78998-d71b-4dec-970d-0fbfa43d2e19'): calling endAction '.
2023-01-09 12:40:25,950+05 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (EE-ManagedThreadFactory-engine-Thread-43674) [] CommandAsyncTask::endCommandAction [within thread] context: Attempting to endAction 'AddImageFromScratch',
2023-01-09 12:40:25,981+05 INFO  [org.ovirt.engine.core.bll.storage.disk.image.AddImageFromScratchCommand] (EE-ManagedThreadFactory-engine-Thread-43674) [e4ee0883-61f7-4940-9545-b2f36734df04] Command [id=b0b78998-d71b-4dec-970d-0fbfa43d2e19]: Updating status to 'FAILED', The command end method logic will be executed by one of its parent commands.
2023-01-09 12:40:25,981+05 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (EE-ManagedThreadFactory-engine-Thread-43674) [e4ee0883-61f7-4940-9545-b2f36734df04] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type 'AddImageFromScratch' completed, handling the result.
2023-01-09 12:40:25,981+05 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (EE-ManagedThreadFactory-engine-Thread-43674) [e4ee0883-61f7-4940-9545-b2f36734df04] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type 'AddImageFromScratch' succeeded, clearing tasks.
2023-01-09 12:40:25,981+05 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engine-Thread-43674) [e4ee0883-61f7-4940-9545-b2f36734df04] SPMAsyncTask::ClearAsyncTask: Attempting to clear task '660f08b4-c380-4ece-86ee-41db40e6bb22'
2023-01-09 12:40:25,982+05 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (EE-ManagedThreadFactory-engine-Thread-43674) [e4ee0883-61f7-4940-9545-b2f36734df04] START, SPMClearTaskVDSCommand( SPMTaskGuidBaseVDSCommandParameters:{storagePoolId='537e6452-8b7a-11ed-a21c-00163e750630', ignoreFailoverLimit='false', taskId='660f08b4-c380-4ece-86ee-41db40e6bb22'}), log id: 4a57fbda
2023-01-09 12:40:25,983+05 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engine-Thread-43674) [e4ee0883-61f7-4940-9545-b2f36734df04] START, HSMClearTaskVDSCommand(HostName = ovirt-host, HSMTaskGuidBaseVDSCommandParameters:{hostId='80024b36-7380-4cf6-936c-9ba01426316f', taskId='660f08b4-c380-4ece-86ee-41db40e6bb22'}), log id: 27ee199b
2023-01-09 12:40:26,036+05 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engine-Thread-43674) [e4ee0883-61f7-4940-9545-b2f36734df04] FINISH, HSMClearTaskVDSCommand, return: , log id: 27ee199b
2023-01-09 12:40:26,036+05 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (EE-ManagedThreadFactory-engine-Thread-43674) [e4ee0883-61f7-4940-9545-b2f36734df04] FINISH, SPMClearTaskVDSCommand, return: , log id: 4a57fbda
2023-01-09 12:40:26,065+05 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engine-Thread-43674) [e4ee0883-61f7-4940-9545-b2f36734df04] BaseAsyncTask::removeTaskFromDB: Removed task '660f08b4-c380-4ece-86ee-41db40e6bb22' from DataBase
2023-01-09 12:40:26,065+05 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (EE-ManagedThreadFactory-engine-Thread-43674) [e4ee0883-61f7-4940-9545-b2f36734df04] CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity 'b0b78998-d71b-4dec-970d-0fbfa43d2e19'
2023-01-09 12:40:27,737+05 INFO  [org.ovirt.engine.core.bll.storage.disk.image.TransferImageStatusCommand] (default task-4) [5deca7ba-3c4e-465c-b64d-853e622e01a4] Running command: TransferImageStatusCommand internal: false. Entities affected :  ID: b0d10f22-f913-4c05-8d44-084f775ae87d Type: SystemAction group CREATE_DISK with role type USER
2023-01-09 12:40:31,739+05 INFO  [org.ovirt.engine.core.bll.storage.disk.image.TransferImageStatusCommand] (default task-4) [83802d31-c7b5-4a80-bf64-af3f3a617d96] Running command: TransferImageStatusCommand internal: false. Entities affected :  ID: b0d10f22-f913-4c05-8d44-084f775ae87d Type: SystemAction group CREATE_DISK with role type USER
2023-01-09 12:40:32,392+05 INFO  [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-26) [e4ee0883-61f7-4940-9545-b2f36734df04] Waiting for disk to be added for image transfer command 'df2c9d30-e452-492d-b07c-b4b1232c9de0'
2023-01-09 12:40:32,393+05 INFO  [org.ovirt.engine.core.bll.storage.disk.AddDiskCommandCallback] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-26) [e4ee0883-61f7-4940-9545-b2f36734df04] Command 'AddDisk' id: '674288b3-6da2-4458-9113-f64afc1e6685' child commands '[b0b78998-d71b-4dec-970d-0fbfa43d2e19]' executions were completed, status 'FAILED'
2023-01-09 12:40:33,418+05 ERROR [org.ovirt.engine.core.bll.storage.disk.AddDiskCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-55) [e4ee0883-61f7-4940-9545-b2f36734df04] Ending command 'org.ovirt.engine.core.bll.storage.disk.AddDiskCommand' with failure.
2023-01-09 12:40:33,451+05 ERROR [org.ovirt.engine.core.bll.storage.disk.image.AddImageFromScratchCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-55) [e4ee0883-61f7-4940-9545-b2f36734df04] Ending command 'org.ovirt.engine.core.bll.storage.disk.image.AddImageFromScratchCommand' with failure.
2023-01-09 12:40:33,493+05 WARN  [org.ovirt.engine.core.bll.storage.disk.AddDiskCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-55) [] VM is null - no unlocking
2023-01-09 12:40:33,609+05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-55) [] EVENT_ID: USER_ADD_DISK_FINISHED_FAILURE(2,022), Add-Disk operation failed to complete.
2023-01-09 12:40:34,644+05 ERROR [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-81) [e4ee0883-61f7-4940-9545-b2f36734df04] Failed to add disk for image transfer command 'df2c9d30-e452-492d-b07c-b4b1232c9de0'
2023-01-09 12:40:35,677+05 ERROR [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-20) [e4ee0883-61f7-4940-9545-b2f36734df04] Ending command 'org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand' with failure.
2023-01-09 12:40:35,677+05 ERROR [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-20) [e4ee0883-61f7-4940-9545-b2f36734df04] Failed to transfer disk '00000000-0000-0000-0000-000000000000' (command id 'df2c9d30-e452-492d-b07c-b4b1232c9de0')
2023-01-09 12:40:35,684+05 WARN  [org.ovirt.engine.core.bll.storage.disk.RemoveDiskCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-20) [78be2ab6] Validation of action 'RemoveDisk' failed for user admin@internal-authz. Reasons: VAR__ACTION__REMOVE,VAR__TYPE__DISK,ACTION_TYPE_FAILED_VM_IMAGE_DOES_NOT_EXIST
2023-01-09 12:40:35,684+05 INFO  [org.ovirt.engine.core.bll.storage.disk.RemoveDiskCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-20) [78be2ab6] Lock freed to object 'EngineLock:{exclusiveLocks='[]', sharedLocks='[]'}'
2023-01-09 12:40:35,684+05 INFO  [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-20) [78be2ab6] Lock freed to object 'EngineLock:{exclusiveLocks='[]', sharedLocks='[]'}'
2023-01-09 12:40:35,685+05 INFO  [org.ovirt.engine.core.bll.storage.disk.image.ImageTransferUpdater] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-20) [78be2ab6] Updating image transfer df2c9d30-e452-492d-b07c-b4b1232c9de0 phase to Finished Failure
2023-01-09 12:40:35,743+05 INFO  [org.ovirt.engine.core.bll.storage.disk.image.TransferImageStatusCommand] (default task-4) [f9eca38b-8cc4-4410-a653-ea091532302a] Running command: TransferImageStatusCommand internal: false. Entities affected :  ID: b0d10f22-f913-4c05-8d44-084f775ae87d Type: SystemAction group CREATE_DISK with role type USER
2023-01-09 12:40:35,827+05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-20) [78be2ab6] EVENT_ID: TRANSFER_IMAGE_FAILED(1,034), Image Upload with disk cirrorstrialpossix2 failed.
2023-01-09 12:40:39,744+05 INFO  [org.ovirt.engine.core.bll.storage.disk.image.TransferImageStatusCommand] (default task-4) [727f4329-71fa-454b-923f-a84f2bfcc7e9] Running command: TransferImageStatusCommand internal: false. Entities affected :  ID: b0d10f22-f913-4c05-8d44-084f775ae87d Type: SystemAction group CREATE_DISK with role type USER

On parsing the logs in the file /var/log/vdsm/vdsm.log on the ovirt-host node, following is the error:

2023-01-09 12:40:20,050+0530 INFO  (tasks/8) [storage.ThreadPool.WorkerThread] START task 660f08b4-c380-4ece-86ee-41db40e6bb22 (cmd=<bound method Task.commit
of <vdsm.storage.task.Task object at 0x7f340c113940>>, args=None) (threadPool:146)
2023-01-09 12:40:20,612+0530 INFO  (tasks/8) [storage.StorageDomain] Create placeholder /rhev/data-center/mnt/[abcd:abcd:abcd::23],[abcd:abcd:abcd::22],[abcd
:abcd:abcd::21]:6789:_volumes_kush_threetest_cb8a69c4-9c9a-4a2d-9a4a-7b9a5ff14c74/b0d10f22-f913-4c05-8d44-084f775ae87d/images/6f649bd6-2235-4ea8-822b-3b47308
b3425 for image's volumes (sd:1718)
2023-01-09 12:40:20,612+0530 ERROR (tasks/8) [storage.TaskManager.Task] (Task='660f08b4-c380-4ece-86ee-41db40e6bb22') Unexpected error (task:877)
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 884, in _run
    return fn(*args, **kargs)
  File "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 350, in run
    return self.cmd(*self.argslist, **self.argsdict)
  File "/usr/lib/python3.6/site-packages/vdsm/storage/securable.py", line 79, in wrapper
    return method(self, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/vdsm/storage/sp.py", line 1952, in createVolume
    initial_size=initialSize, add_bitmaps=addBitmaps)
  File "/usr/lib/python3.6/site-packages/vdsm/storage/sd.py", line 1216, in createVolume
    initial_size=initial_size, add_bitmaps=add_bitmaps)
  File "/usr/lib/python3.6/site-packages/vdsm/storage/volume.py", line 1174, in create
    imgPath = dom.create_image(imgUUID)
  File "/usr/lib/python3.6/site-packages/vdsm/storage/sd.py", line 1721, in create_image
    "create_image_rollback", [image_dir])
  File "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 385, in __init__
    self.params = ParamList(argslist)
  File "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 298, in __init__
    raise ValueError("ParamsList: sep %s in %s" % (sep, i))
ValueError: ParamsList: sep , in /rhev/data-center/mnt/[abcd:abcd:abcd::23],[abcd:abcd:abcd::22],[abcd:abcd:abcd::21]:6789:_volumes_kush_threetest_cb8a69c4-9
c9a-4a2d-9a4a-7b9a5ff14c74/b0d10f22-f913-4c05-8d44-084f775ae87d/images/6f649bd6-2235-4ea8-822b-3b47308b3425
2023-01-09 12:40:21,530+0530 INFO  (tasks/8) [storage.ThreadPool.WorkerThread] FINISH task 660f08b4-c380-4ece-86ee-41db40e6bb22 (threadPool:148)
2023-01-09 12:40:21,715+0530 INFO  (jsonrpc/4) [api.host] START getAllVmStats() from=::1,52540 (api:48)
2023-01-09 12:40:21,715+0530

Kindly help us out.
Because of this we are unable to create a disk. and the disks status is illegal.

Thanks and Regards