allow keeping input folder during job cleanup

[vieting]

When sisyphus cleans up a job folder, the input folder is deleted. This can be impractical if you want to find some job's dependencies because instead of just going through the input folders, you need to open each finished.tar.gz file and check for input jobs in the info file. I therefore propose to add the config option JOB_CLEANUP_KEEP_INPUT. It set to True, the input folder will remain even after cleaning.

Edit: As suggested in the PR, we set JOB_CLEANUP_KEEP_INPUT = True as the default setting. This changes the default behavior, but is not expected to hurt anyone.

[michelwi]

This changes the default behavior from "add input folder to tar archive" to "delete input folder". But I even slightly prefer the new behavior.

[vieting]

This changes the default behavior from "add input folder to tar archive" to "delete input folder". But I even slightly prefer the new behavior.

Right, should have mentioned this in the description. I'd expect that an input folder in the tar archive is hardly ever used. But we can also keep it in there if others prefer that.

[albertz]

you need to open each finished.tar.gz file and check for input jobs in the info file

I'm not sure I fully understand. You don't need to open the finished.tar.gz. You can just open the info file directly, which is not removed?

[vieting]

you need to open each finished.tar.gz file and check for input jobs in the info file

I'm not sure I fully understand. You don't need to open the finished.tar.gz. You can just open the info file directly, which is not removed?

Right, the info file is not inside the tar archive. Still, I prefer to easily step through the input dependencies by going through the input folders. This is way easier than checking the input in the info file, then going to that job's info file and so forth.

[albertz]

I would mention in the description of the commit and PR that this changes the default behavior. But I'm fine with that.

I'm also fine with the other changes otherwise.