[Bug]: Smoke test jenkins job failed to clean up fo RPM and DEB

[zelinh]

Describe the bug

When running the smoke tests Jenkins job on either RPM or DEB distribution, the tests themselves can be running correctly. However, when tests completed, Jenkins failed to clean up the workspace.

Seeing error

[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is disabled by the job configuration...
ERROR: Cannot delete workspace :Unable to delete '/var/jenkins/workspace/smoke-test/test-results'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts.
[Pipeline] }
[Pipeline] // script
Post stage
[Pipeline] cleanWs
[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is disabled by the job configuration...
ERROR: Cannot delete workspace :Unable to delete '/var/jenkins/workspace/smoke-test/test-results'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts.
Error when executing always post condition:
Also:   org.jenkinsci.plugins.workflow.actions.ErrorAction$ErrorId: 2269c912-143e-4975-8742-7190e98bbe63
hudson.AbortException: Cannot delete workspace: Unable to delete '/var/jenkins/workspace/smoke-test/test-results'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts.
	at hudson.plugins.ws_cleanup.WsCleanup.perform(WsCleanup.java:249)
	at jenkins.tasks.SimpleBuildStep.perform(SimpleBuildStep.java:123)
	at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:101)
	at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:71)
	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)

To reproduce

Start the smoke tests Jenkins job with parameters

TEST_MANIFEST: 2.19.0/opensearch-2.19.0-test.yml
BUILD_MANIFEST_URL: https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.19.0/10691/linux/x64/rpm/builds/opensearch/manifest.yml

[zelinh]

After investigation, I believe the issue is about the root user we specify for the docker.

Both RPM & DEB requires root user access to start the cluster so we enable -u root when running docker container.
During runtime, smoke test workflow within docker container generates /var/jenkins/workspace/smoke-test/test-results with docker root user; while after test workflow completes the Jenkins user is not able to delete those files and cause the error.

I will try to do a fix as we don't have to use root user to start the test workflow.