sct docker backend deletes all docker instances on the system in teardown

[asias]

It should only delete instances started by sct test, not all instances on the system.

< t:2024-10-23 02:04:54,597 f:remote_base.py  l:560  c:RemoteLibSSH2CmdRunner p:DEBUG > <172.18.0.5>: Running command "docker ps -a -q | xargs docker rm -f"...

[asias]

Got another example:

< t:2024-10-23 07:00:20,328 f:cluster.py      l:1998 c:sdcm.cluster_docker  p:INFO  > Node perf-latency-nemesis-asias-monitor-node-da4ee3ad-0 [127.0.0.1 | 127.0.0.1]: Installing Scylla...                                                                                                                                                                                                                                                                                   We trust you have received the usual lecture from the local System                                                                                                                                                                     Administrator. It usually boils down to these three things:                                                                                                                                                                                                                                                                                                                                                                                                                       #1) Respect the privacy of others.                                                                                                                                                                                                     #2) Think before you type.                                                                                     
    #3) With great power comes great responsibility.        

[fruch]

this one goes way back when YCSB was introduced for alternator

f971bb9

that code was written assuming it runs on a loader node, so docker backend running locally wasn't considered.

we should overwrite kill_docker_loaders for docker backend, and kill more selectively based on the test_id label we should have to the docker run (and and we are missing them, we should introduce those labels for the stress docker instances)

[dimakr]

@fruch
Not sure that the command docker ps -a -q | xargs docker rm -f deletes containers on the system as the only place where it is executed is

scylla-cluster-tests/sdcm/cluster.py

Line 5274 in 1434c81

loader.remoter.run(cmd='docker ps -a -q | xargs docker rm -f', verbose=True, ignore_status=True)

I.e. in case of docker backend it is requested to be executed in a loader container, not in the on the SCT runner (local host).

[fruch]

@fruch
Not sure that the command docker ps -a -q | xargs docker rm -f deletes containers on the system as the only place where it is executed is

scylla-cluster-tests/sdcm/cluster.py

Line 5274 in 1434c81

loader.remoter.run(cmd='docker ps -a -q | xargs docker rm -f', verbose=True, ignore_status=True)

I.e. in case of docker backend it is requested to be executed in a loader container, not in the on the SCT runner (local host).

There's one docker engine on the host, regardless where you give the command from.

So when using docker backend this would clear all of the docker instances

[dimakr]

So when using docker backend this would clear all of the docker instances

Ok. I was not able to reproduce it, but will give another try.

[dimakr]

@asias Could you please share details/command of how you was doing the problematic SCT run against docker backend (was it via hydra, ./sct.py, etc.)?

[asias]

@asias Could you please share details/command of how you was doing the problematic SCT run against docker backend (was it via hydra, ./sct.py, etc.)?

e.g.,

hydra run-test performance_regression_test.PerformanceRegressionTest.test_latency_write_with_nemesis --backend docker --config test-cases/performance/perf-regression-latency-650gb-with-nemesis.yaml --config configurations/tablets_disabled.yaml --config configurations/disable_kms.yaml