jdk_container left files owned by root

[llxia]

jdk_container left files on the host machine that are owned by root. These files cannot be cleaned by Jenkins job. It causes Jenkins job to fail.

12:05:52  ERROR: Cannot delete workspace :Unable to delete '/home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17113857837219/jdk_container_0/work/scratch/2/jdk-sharedtmp/.com_ibm_tools_attach/_controller'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts.
[Pipeline] echo
12:05:52  Exception: hudson.AbortException: Cannot delete workspace: Unable to delete '/home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17113857837219/jdk_container_0/work/scratch/2/jdk-sharedtmp/.com_ibm_tools_attach/_controller'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts.
[Pipeline] sh
12:05:53  + rm -rf /home/jenkins/workspace/Grinder/aqa-tests/TKG
12:05:53  rm: cannot remove '/home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17113857837219/jdk_container_0/work/scratch/2/jdk-sharedtmp/.com_ibm_tools_attach/_controller': Operation not permitted
12:05:53  rm: cannot remove '/home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17113857837219/jdk_container_0/work/scratch/2/jdk-sharedtmp/.com_ibm_tools_attach/_notifier': Operation not permitted

@sophia-guo @smlambert do you also see a similar issue at Adoptium Jenkins? Is there a better way to resolve this?

[sophia-guo]

These tests were added around one and half years ago. As it's dev level may not run frequently. I didn't notice there is this issue. Check recent jdk21 seems no this issue.

https://ci.adoptium.net/view/Test_openjdk/job/Test_openjdk21_hs_dev.openjdk_x86-64_linux/

[llxia]

We should mark the node offline automatically when there is an error Cannot delete workspace: Unable to delete ...

[AswathySK]

Is there any other way to clean up the crash files in the test code itself instead of marking it offline @llxia

[smlambert]

Is there any other way to clean up the crash files in the test code itself instead of marking it offline @llxia

llxia is on vacation

Related:
https://stackoverflow.com/questions/42423999/cant-delete-file-created-via-docker

[sophia-guo]

I think we also need to know why this happens. Does it only happen when impl=openj9|ibm as no issue reported with jdk_container running against impl=hotspot.

Normally this permission issue happens if you run things as root inside the container while using a mapped volume from the host inside the container. The jdk_container tests map volumes options are like --volume /home/jenkins/workspace/jenkinsjobname/aqa-tests/TKG/output_***/jdk_container_0/work/classes/2/...., which is not the workdir. So shouldn't have this issue. Is there something specific to openj9|ibm caused this?

[AswathySK]

Is there any updates on this issue?
Is @llxia back from vacation?

[smlambert]

I think we also need to know why this happens. Does it only happen when impl=openj9|ibm as no issue reported with jdk_container running against impl=hotspot.

If I had to guess, it happens when a testcase fails and doesn't cleanup after itself, then the workspace can not be deleted. So @AswathySK perhaps check if that is the case and exclude the failing testcases.

Lan is not back from vacation and no one is pursuing this issue further at this time. I suggest you dig in to answer some of the questions in this issue if you are interested in a different approach than taking the machine offline.

[sophia-guo]

Just a note that PR of making the node offine has also been reverted, which might help @AswathySK your investigation?

[AswathySK]

@smlambert , when a test case fails it is not able to clean up after since the files created when it crashes are owned by root user. And yes I will do some more investigation on which all test cases we are seeing this issue.

[smlambert]

So my point is, the reason we do not have a cleanup problem for Temurin is that there is not a failing/crashing testcase.

So your first task would be to see which testcase is crashing/failing, triage it by gathering any extra data you can, report the issue in the openj9 repo if it doesn't already exist, and exclude the test in the ProblemList files while the issue is being investigated and fixed by the openj9 team.

[AdamBrousseau]

Does the cleanWs() happen inside the container or after the container is exited? Still a bit odd that it is only a few files that are owned by root. 🤔

[llxia]

cleanWs() (at groovy level) happens after the jdk_container is exited. jdk_container is a openjdk tests which does not belong to us.

[AdamBrousseau]

Has anyone tried removing the files as the jenkins user via any random container as the root user? As much as that could be a hack workaround, it is probably easy to add a step to our cleanup scripts to remove everything inside the workspace dir via a container. cc @AswathySK