Parallelize the `provision` step #2468

happz · 2023-11-09T12:38:30Z

This change requires refactoring of current queue internals, to better support the third kind of parallelization & queueing. A custom queue task is needed, to run provisioning in parallel, and the step takes care of ordering of provision phases and actions.

Implements #2244

Pull Request Checklist

implement the feature
include a release note

lukaszachy · 2023-11-21T16:36:57Z

Hm, I've tried with virtual and it doesn't work well.

/test:
    test: echo
    
/plan:
    execute:
        how: tmt
    discover:
        how: fmf
        
    provision:
        - how: virtual
        - how: virtual
        - how: virtual

$ tmt run --until report
/var/tmp/tmt/run-117

/p/plan
    discover
        how: fmf
        directory: /tmp/r_173145_H6I
        summary: 1 test selected
    provision
        queued provision task #1: default-0, default-1 and default-2
        
        provision task #1: default-0, default-1 and default-2
[default-0]         started
[default-0]         how: virtual
[default-1]         started
[default-0]         memory: 2048 megabyte
[default-0]         disk: 40 gigabyte
[default-1]         how: virtual
[default-2]         started
[default-1]         memory: 2048 megabyte
[default-1]         disk: 40 gigabyte
[default-2]         how: virtual
[default-2]         memory: 2048 megabyte
[default-2]         disk: 40 gigabyte
[default-0]         progress: booting...
[default-1]         progress: booting...
[default-2]         progress: booting...
libvirt: QEMU Driver error : internal error: QEMU unexpectedly closed the monitor (vm='tmt-117-idTZSMZI'): 2023-11-21T16:35:32.338285Z qemu-system-x86_64: -netdev user,id=testcloud_net.10042,hostfwd=tcp::10042-:22: Could not set up host forwarding rule 'tcp::10042-:22'
Instance startup failed, retrying in 5 seconds...
libvirt: QEMU Driver error : internal error: QEMU unexpectedly closed the monitor (vm='tmt-117-idTZSMZI'): 2023-11-21T16:35:37.575292Z qemu-system-x86_64: -netdev user,id=testcloud_net.10042,hostfwd=tcp::10042-:22: Could not set up host forwarding rule 'tcp::10042-:22'
[default-0]         finished
[default-0]         fail: Failed to boot testcloud instance (internal error: QEMU unexpectedly closed the monitor (vm='tmt-117-idTZSMZI'): 2023-11-21T16:35:37.575292Z qemu-system-x86_64: -netdev user,id=testcloud_net.10042,hostfwd=tcp::10042-:22: Could not set up host forwarding rule 'tcp::10042-:22').
[default-2]         finished
[default-2]         multihost name: default-2
[default-2]         arch: x86_64
[default-2]         distro: Fedora Linux 39 (Cloud Edition)
[default-1]         finished
[default-1]         multihost name: default-1
[default-1]         arch: x86_64
[default-1]         distro: Fedora Linux 39 (Cloud Edition)

plan failed

The exception was caused by 1 earlier exceptions

Cause number 1:

    provision step failed

    The exception was caused by 1 earlier exceptions

    Cause number 1:

        Failed to boot testcloud instance (internal error: QEMU unexpectedly closed the monitor (vm='tmt-117-idTZSMZI'): 2023-11-21T16:35:37.575292Z qemu-system-x86_64: -netdev user,id=testcloud_net.10042,hostfwd=tcp::10042-:22: Could not set up host forwarding rule 'tcp::10042-:22').

There should be 3 machines running but virsh shows that one of them is paused.

tmt/steps/prepare/__init__.py

happz · 2023-11-21T17:19:30Z

Hm, I've tried with virtual and it doesn't work well.

...
libvirt: QEMU Driver error : internal error: QEMU unexpectedly closed the monitor (vm='tmt-117-idTZSMZI'): 2023-11-21T16:35:32.338285Z qemu-system-x86_64: -netdev user,id=testcloud_net.10042,hostfwd=tcp::10042-:22: Could not set up host forwarding rule 'tcp::10042-:22'
Instance startup failed, retrying in 5 seconds...
libvirt: QEMU Driver error : internal error: QEMU unexpectedly closed the monitor (vm='tmt-117-idTZSMZI'): 2023-11-21T16:35:37.575292Z qemu-system-x86_64: -netdev user,id=testcloud_net.10042,hostfwd=tcp::10042-:22: Could not set up host forwarding rule 'tcp::10042-:22'
[default-0] finished
[default-0] fail: Failed to boot testcloud instance (internal error: QEMU unexpectedly closed the monitor (vm='tmt-117-idTZSMZI'): 2023-11-21T16:35:37.575292Z qemu-system-x86_64: -netdev user,id=testcloud_net.10042,hostfwd=tcp::10042-:22: Could not set up host forwarding rule 'tcp::10042-:22').
[default-2] finished
[default-2] multihost name: default-2
[default-2] arch: x86_64
[default-2] distro: Fedora Linux 39 (Cloud Edition)
[default-1] finished
[default-1] multihost name: default-1
[default-1] arch: x86_64
[default-1] distro: Fedora Linux 39 (Cloud Edition)

plan failed

The exception was caused by 1 earlier exceptions

Cause number 1:
provision step failed

The exception was caused by 1 earlier exceptions

Cause number 1:

    Failed to boot testcloud instance (internal error: QEMU unexpectedly closed the monitor (vm='tmt-117-idTZSMZI'): 2023-11-21T16:35:37.575292Z qemu-system-x86_64: -netdev user,id=testcloud_net.10042,hostfwd=tcp::10042-:22: Could not set up host forwarding rule 'tcp::10042-:22').
There should be 3 machines running but virsh shows that one of them is paused.

Worked like a charm on my laptop :/ Can you check whether there's some libvirt/qemu/testcloud logging related to this? I'm not versed in Qemu things, but it looks like some kind of conflict. @frantisekz any idea what might be we hitting?

lukaszachy · 2023-11-21T20:43:58Z

@happz I'd say https://pagure.io/testcloud/blob/master/f/testcloud/util.py#_60 is not ready for parallel requests.

frantisekz · 2023-11-21T21:04:13Z

I'll take a look early December (currently on PTO without laptop).

happz · 2023-11-22T08:39:38Z

@happz I'd say https://pagure.io/testcloud/blob/master/f/testcloud/util.py#_60 is not ready for parallel requests.

Right, that might be the case, easily.

~~Can you try once more with the following patch? It should prevent concurrent calls into testcloud library:~~

Never mind, added it as a commit to this PR.

qcheng-redhat · 2023-11-24T10:27:06Z

I tried the latest patch with libvirt-provision plugin, it works. I got three guests provisioned parallelized.

My plan:

execute:
how: tmt

/plan:
provision:
- how: libvirt
guestargs:
--cpu host-model
--ram=4096
--vcpus=2
--graphics vnc,listen=0.0.0.0
distroname: xxxxxx
- how: libvirt
guestargs:
--cpu host-model
--ram=4096
--vcpus=2
--graphics vnc,listen=0.0.0.0
distroname: xxxxxx
- how: libvirt
guestargs:
--cpu host-model
--ram=4096
--vcpus=2
--graphics vnc,listen=0.0.0.0
distroname: xxxxxx
discover:
how: fmf

log:
04:47:17 queued provision task #1: default-0, default-1 and default-2
04:47:17
04:47:17 provision task #1: default-0, default-1 and default-2
04:47:17 [default-0] started
04:47:17 [default-0] how: libvirt
04:47:17 [default-1] started
04:47:17 [default-0] order: 50
04:47:17 [default-1] how: libvirt
04:47:17 [default-2] started
04:47:17 [default-1] order: 50
04:47:17 [default-2] how: libvirt
04:47:17 [default-2] order: 50
...
04:54:53
04:54:53 summary: 3 guests provisioned

happz · 2023-11-24T12:44:18Z

I tried the latest patch with libvirt-provision plugin, it works. I got three guests provisioned parallelized.

Glad to hear that, thanks!

/plan: provision: - how: libvirt guestargs: --cpu host-model --ram=4096 --vcpus=2 --graphics vnc,listen=0.0.0.0 distroname: xxxxxx - how: libvirt guestargs: --cpu host-model --ram=4096 --vcpus=2 --graphics vnc,listen=0.0.0.0 distroname: xxxxxx - how: libvirt guestargs: --cpu host-model --ram=4096 --vcpus=2 --graphics vnc,listen=0.0.0.0 distroname: xxxxxx discover: how: fmf

On another note: please, consider supporting hardware field, https://tmt.readthedocs.io/en/stable/spec/hardware.html, at least for the already supported requirements, cpu and memory. Plugins shipped with tmt would express part of the guestargs field from your example like this:

hardware:
  cpu:
    cores: 2
  memory: 4096 MB

I know it's probably of no immediate benefit to you now, and I would fully understand the hesitation or focus on more needed features for your use cases, but in the long run, it would help end-users if plugins would speak the same language. The field is supported by tmt's own plugins, by Testing Farm, and we're slowly extending the requirements supported by individual plugins. I suppose your plugin is very similar to virtual in tmt repo, it would transform the requirements from hardware field into options, e.g. https://github.com/teemtee/tmt/blob/main/tmt/steps/provision/testcloud.py#L489 demonstrates how we did this for virtual plugin. I'd be happy to provide any guidance if you decide t oextend your plugin.

qcheng-redhat · 2023-11-27T01:54:19Z

I tried the latest patch with libvirt-provision plugin, it works. I got three guests provisioned parallelized.

Glad to hear that, thanks!

/plan: provision: - how: libvirt guestargs: --cpu host-model --ram=4096 --vcpus=2 --graphics vnc,listen=0.0.0.0 distroname: xxxxxx - how: libvirt guestargs: --cpu host-model --ram=4096 --vcpus=2 --graphics vnc,listen=0.0.0.0 distroname: xxxxxx - how: libvirt guestargs: --cpu host-model --ram=4096 --vcpus=2 --graphics vnc,listen=0.0.0.0 distroname: xxxxxx discover: how: fmf

On another note: please, consider supporting hardware field, https://tmt.readthedocs.io/en/stable/spec/hardware.html, at least for the already supported requirements, cpu and memory. Plugins shipped with tmt would express part of the guestargs field from your example like this:
hardware:
  cpu:
    cores: 2
  memory: 4096 MB
I know it's probably of no immediate benefit to you now, and I would fully understand the hesitation or focus on more needed features for your use cases, but in the long run, it would help end-users if plugins would speak the same language. The field is supported by tmt's own plugins, by Testing Farm, and we're slowly extending the requirements supported by individual plugins. I suppose your plugin is very similar to virtual in tmt repo, it would transform the requirements from hardware field into options, e.g. https://github.com/teemtee/tmt/blob/main/tmt/steps/provision/testcloud.py#L489 demonstrates how we did this for virtual plugin. I'd be happy to provide any guidance if you decide t oextend your plugin.

Hi @happz ,

Thanks for the suggestion of using TMT hardware specification and sharing the virtual plugin example. Let's discuss this topic offline.

Thanks,
Qinghua

psss · 2023-11-28T08:52:59Z

I've tried the parallel provision with the /tests/multihost/web test several times and it always failed with:

Failed to connect in 60s.

Twice it was just a single box which failed and once even two of them. @happz, have you seen anything like this?

happz · 2023-11-28T08:58:09Z

I've tried the parallel provision with the /tests/multihost/web test several times and it always failed with:
Failed to connect in 60s.
Twice it was just a single box which failed and once even two of them. @happz, have you seen anything like this?

Nope, I haven't seen this error.

Well, I'm afraid we should push this one out into 1.31. The PR is changing the whole step, you're hitting issues I haven't seen, there's barely any review, and it's Nov 28th :/

happz · 2023-12-05T15:50:35Z

/packit test --identifier full

tmt/queue.py

tmt/steps/__init__.py

tmt/steps/provision/__init__.py

psss · 2024-01-04T08:42:22Z

I've tried the parallel provision with the /tests/multihost/web test several times and it always failed with:
Failed to connect in 60s.
Twice it was just a single box which failed and once even two of them. @happz, have you seen anything like this?
Nope, I haven't seen this error.

Interesting. Tried today again and encountered similar behaviour. Seems that the problem happens randomly. Run four times:

httpd-server failed to connect in 60s.
httpd-server failed to connect in 60s.
curl-client failed to connect in 60s.
everything fine & tests passed

Looking through the log I see:

err: Warning: Permanently added '192.168.124.110' (ED25519) to the list of known hosts.
err: ssh_askpass: exec(/usr/libexec/openssh/ssh-askpass): No such file or directory
err: Permission denied, please try again.
err: ssh_askpass: exec(/usr/libexec/openssh/ssh-askpass): No such file or directory
err: Permission denied, please try again.
err: ssh_askpass: exec(/usr/libexec/openssh/ssh-askpass): No such file or directory
err: Received disconnect from 192.168.124.110 port 22:2: Too many authentication failures
err: Disconnected from 192.168.124.110 port 22

Which is weird because the command line is the same as for the other guests:

Run command: ssh -oForwardX11=no -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oServerAliveInterval=60 -oServerAliveCountMax=5 -oIdentitiesOnly=yes -p22 -i /var/tmp/tmt/run-024/plan/provision/curl-client/id_ecdsa -S/run/user/12559/tmt/tmpyjjy9wtt [email protected] 'export TMT_PLAN_DATA=/var/tmp/tmt/run-024/plan/data; export TMT_PLAN_ENVIRONMENT_FILE=/var/tmp/tmt/run-024/plan/data/variables.env; export TMT_TREE=/var/tmp/tmt/run-024/plan/tree; export TMT_VERSION=1.30.0.dev64+g9df4fee9.d20240103; whoami'

Any idea what might be wrong?

happz · 2024-01-04T08:46:58Z

I've tried the parallel provision with the /tests/multihost/web test several times and it always failed with:
Failed to connect in 60s.
Twice it was just a single box which failed and once even two of them. @happz, have you seen anything like this?
Nope, I haven't seen this error.
Interesting. Tried today again and encountered similar behaviour. Seems that the problem happens randomly. Run four times:

httpd-server failed to connect in 60s.

httpd-server failed to connect in 60s.

curl-client failed to connect in 60s.

everything fine & tests passed

Looking through the log I see:
err: Warning: Permanently added '192.168.124.110' (ED25519) to the list of known hosts.
err: ssh_askpass: exec(/usr/libexec/openssh/ssh-askpass): No such file or directory
err: Permission denied, please try again.
err: ssh_askpass: exec(/usr/libexec/openssh/ssh-askpass): No such file or directory
err: Permission denied, please try again.
err: ssh_askpass: exec(/usr/libexec/openssh/ssh-askpass): No such file or directory
err: Received disconnect from 192.168.124.110 port 22:2: Too many authentication failures
err: Disconnected from 192.168.124.110 port 22
Which is weird because the command line is the same as for the other guests:
Run command: ssh -oForwardX11=no -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oServerAliveInterval=60 -oServerAliveCountMax=5 -oIdentitiesOnly=yes -p22 -i /var/tmp/tmt/run-024/plan/provision/curl-client/id_ecdsa -S/run/user/12559/tmt/tmpyjjy9wtt [email protected] 'export TMT_PLAN_DATA=/var/tmp/tmt/run-024/plan/data; export TMT_PLAN_ENVIRONMENT_FILE=/var/tmp/tmt/run-024/plan/data/variables.env; export TMT_TREE=/var/tmp/tmt/run-024/plan/tree; export TMT_VERSION=1.30.0.dev64+g9df4fee9.d20240103; whoami'
Any idea what might be wrong?

Stinks like a race condition, threads contesting shared resources or something similar. I'll try to debug it in the afternoon.

psss · 2024-01-04T09:13:55Z

Stinks like a race condition, threads contesting shared resources or something similar. I'll try to debug it in the afternoon.

Thanks. The detailed debug output from ssh looks like this:

debug1: Offering public key: /home/psss/.ssh/id_rsa RSA SHA256:OY7RDp7mZHS6OzXvc9hnQk2qaDCcOuFM2iaGD8GOrKc explicit agent
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Offering public key: /home/psss/.ssh/1minutetip_rsa RSA SHA256:9j1blwt3wcrRiGYZQ7ZGu9axm3cDklH6/z4c+Ee8CzE explicit agent
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Offering public key: /var/tmp/tmt/run-031/plan/provision/wget-client/id_ecdsa ECDSA SHA256:i3ivQgMXnx8vh5PJ/L1K0LKqIs5RToHLZqA+d0QCbtM explicit
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug2: we did not send a packet, disable method
debug3: authmethod_lookup password
debug3: remaining preferred: ,password
debug3: authmethod_is_enabled password
debug1: Next authentication method: password
...

Seems like the provided key is not accepted.

happz · 2024-01-04T09:49:04Z

@psss hopefully addressed in 87ec95f: added more locking in testcloud plugin - not all calls into testcloud library were protected, and added _thread_safe flag for provisioning plugins. Since we don't control all libraries used by our provisioning plugins, we get more time to verify they do work when used in parallel provisioning.

psss

Thanks much for implementing this! Looks good and works nicely. Added just some minor questions and suggestions.

tmt/queue.py

tmt/steps/provision/__init__.py

psss · 2024-01-04T18:33:22Z

This definitely deserves a mention in the release notes. Also, what do you think about adding a short paragraph with the overview of steps which support parallel execution to the Guide?

happz · 2024-01-04T21:06:50Z

This definitely deserves a mention in the release notes.

Sure, added one in a8d369a.

Also, what do you think about adding a short paragraph with the overview of steps which support parallel execution to the Guide?

I mentioned compatible plugins in the release note, and I was hoping to make this information available on plugin pages, once #2549 lands

psss · 2024-01-05T10:05:32Z

Also, what do you think about adding a short paragraph with the overview of steps which support parallel execution to the Guide?

I mentioned compatible plugins in the release note, and I was hoping to make this information available on plugin pages, once #2549 lands

Makes sense, thanks.

psss

Thanks for addressing all issues. Looks good now. I've just slightly modified the release note formatting in 59681ab.

psss · 2024-01-05T13:21:23Z

Many tests are failing because of:

PullTask.__init__() got an unexpected keyword argument 'result'

happz · 2024-01-05T14:34:32Z

/packit test --identifier full

Many tests are failing because of:
PullTask.__init__() got an unexpected keyword argument 'result'

Adding b6a3250, let's see whether it helps.

psss · 2024-01-05T15:05:42Z

/packit test --identifier full

This change requires refactoring of current queue internals, to better support the third kind of parallelization & queueing. A custom queue task is needed, to run provisioning in parallel, and the step takes care of ordering of provision phases and actions.

psss · 2024-01-05T17:29:26Z

/packit test --identifier full

guoguojenna · 2024-01-12T03:27:08Z

Seems beaker provisioning is still in sequence, any reasons?

Thanks.

guoguojenna · 2024-01-12T11:27:43Z

tmt/steps/provision/mrack.py

@@ -680,6 +680,8 @@ class ProvisionBeaker(tmt.steps.provision.ProvisionPlugin[ProvisionBeakerData]):
    _data_class = ProvisionBeakerData
    _guest_class = GuestBeaker

+    # _thread_safe = True


Uncomment it, use _thread_safe = True, beaker parallel provisioning works for me. Wondering why you comment it?

[client-1] guest: has been requested
[client-1] job id: 8784929
[client-1] wait: waiting for condition 'get_new_state' with timeout 1:00:00, deadline in 3600.0 seconds, checking every 60.00 seconds
[server-1] guest: has been requested
[server-1] job id: 8784930
[server-1] wait: waiting for condition 'get_new_state' with timeout 1:00:00, deadline in 3600.0 seconds, checking every 60.00 seconds
[client-2] guest: has been requested
[client-2] job id: 8784931
...
summary: 3 guests provisioned
client-2
client-1
server-1

Because we are not sure whether the library tmt uses for Beaker provisioning, mrack, is thread-safe. We did run into issues with the virtual plugin, where the problem was eventually solved by adding an extra lock to provide serialization, but there was no time invested into doing the same for the Beaker plugin. If it's required, that is.

Filed #2607 to track this.

happz added step | provision Stuff related to the provision step area | multihost Issues related to the multihost testing support labels Nov 9, 2023

happz added this to the 1.30 milestone Nov 9, 2023

happz force-pushed the parallelize-provision branch 2 times, most recently from 3c87330 to 72c73fd Compare November 14, 2023 14:02

happz marked this pull request as ready for review November 14, 2023 14:02

happz requested review from psss, lukaszachy, thrix and janhavlin as code owners November 14, 2023 14:02

lukaszachy reviewed Nov 21, 2023

View reviewed changes

tmt/steps/prepare/__init__.py Outdated Show resolved Hide resolved

happz force-pushed the parallelize-provision branch from 72c73fd to 94ec544 Compare November 21, 2023 17:19

psss changed the title ~~Parallelize provisioning~~ Parallelize the provision step Nov 22, 2023

happz requested a review from frantisekz as a code owner November 22, 2023 08:44

psss linked an issue Nov 28, 2023 that may be closed by this pull request

Parallelize the provision step #2244

Closed

psss modified the milestones: 1.30, 1.31 Nov 28, 2023

happz force-pushed the parallelize-provision branch from db099f3 to f554c5a Compare December 5, 2023 10:21

happz force-pushed the parallelize-provision branch from f554c5a to e13018a Compare December 5, 2023 19:43

KwisatzHaderach reviewed Jan 3, 2024

View reviewed changes

tmt/queue.py Outdated Show resolved Hide resolved

tmt/steps/__init__.py Outdated Show resolved Hide resolved

tmt/steps/provision/__init__.py Show resolved Hide resolved

happz force-pushed the parallelize-provision branch from ac12208 to 0b21eb3 Compare January 3, 2024 22:16

happz force-pushed the parallelize-provision branch from 0b21eb3 to 87ec95f Compare January 4, 2024 09:46

happz requested a review from dav-pascual as a code owner January 4, 2024 09:46

KwisatzHaderach approved these changes Jan 4, 2024

View reviewed changes

psss requested changes Jan 4, 2024

View reviewed changes

psss self-assigned this Jan 4, 2024

happz force-pushed the parallelize-provision branch from 87ec95f to db0510d Compare January 4, 2024 20:51

psss approved these changes Jan 5, 2024

View reviewed changes

happz force-pushed the parallelize-provision branch from 59681ab to 5a07401 Compare January 5, 2024 10:45

psss force-pushed the parallelize-provision branch from 5a07401 to 0aa7db6 Compare January 5, 2024 10:54

psss force-pushed the parallelize-provision branch from 4a5e340 to 8196422 Compare January 5, 2024 17:18

psss merged commit 8196422 into main Jan 5, 2024
18 checks passed

psss deleted the parallelize-provision branch January 5, 2024 20:02

guoguojenna reviewed Jan 12, 2024

View reviewed changes

happz mentioned this pull request Jan 12, 2024

Investigate whether beaker plugin could be made thread-safe #2607

Open

psss mentioned this pull request Apr 10, 2024

Support multihost tests #726

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Parallelize the `provision` step #2468

Parallelize the `provision` step #2468

happz commented Nov 9, 2023 •

edited by psss

Loading

lukaszachy commented Nov 21, 2023

happz commented Nov 21, 2023

lukaszachy commented Nov 21, 2023

frantisekz commented Nov 21, 2023

happz commented Nov 22, 2023 •

edited

Loading

qcheng-redhat commented Nov 24, 2023

happz commented Nov 24, 2023

qcheng-redhat commented Nov 27, 2023

psss commented Nov 28, 2023

happz commented Nov 28, 2023

happz commented Dec 5, 2023

psss commented Jan 4, 2024

happz commented Jan 4, 2024

psss commented Jan 4, 2024

happz commented Jan 4, 2024

psss left a comment

psss commented Jan 4, 2024

happz commented Jan 4, 2024

psss commented Jan 5, 2024

psss left a comment

psss commented Jan 5, 2024

happz commented Jan 5, 2024

psss commented Jan 5, 2024

psss commented Jan 5, 2024

guoguojenna commented Jan 12, 2024

guoguojenna Jan 12, 2024

happz Jan 12, 2024

Parallelize the provision step #2468

Parallelize the provision step #2468

Conversation

happz commented Nov 9, 2023 • edited by psss Loading

lukaszachy commented Nov 21, 2023

happz commented Nov 21, 2023

lukaszachy commented Nov 21, 2023

frantisekz commented Nov 21, 2023

happz commented Nov 22, 2023 • edited Loading

qcheng-redhat commented Nov 24, 2023

happz commented Nov 24, 2023

qcheng-redhat commented Nov 27, 2023

psss commented Nov 28, 2023

happz commented Nov 28, 2023

happz commented Dec 5, 2023

psss commented Jan 4, 2024

happz commented Jan 4, 2024

psss commented Jan 4, 2024

happz commented Jan 4, 2024

psss left a comment

Choose a reason for hiding this comment

psss commented Jan 4, 2024

happz commented Jan 4, 2024

psss commented Jan 5, 2024

psss left a comment

Choose a reason for hiding this comment

psss commented Jan 5, 2024

happz commented Jan 5, 2024

psss commented Jan 5, 2024

psss commented Jan 5, 2024

guoguojenna commented Jan 12, 2024

guoguojenna Jan 12, 2024

Choose a reason for hiding this comment

happz Jan 12, 2024

Choose a reason for hiding this comment

Parallelize the `provision` step #2468

Parallelize the `provision` step #2468

happz commented Nov 9, 2023 •

edited by psss

Loading

happz commented Nov 22, 2023 •

edited

Loading