Jail cleanup issue.

[mmeeks]

Share profile as of now:

6% of server time in that statting - which seems amazing; perhaps there is some explosion of crashing kits or something - is it possible that: if (time(nullptr) < modifiedTimeSec + 180)
continue;
doens't work - and we build a huge list of crashed kits - but the; kits should not crash on share much - a tad perplexed by that =)
ls /opt/cool/child-roots/873022-20fbac79/ | wc -l
742
interesting.
vs. 64 coolforkit processes

It looks like we are leaking jails.

[mmeeks]

[pid 873027] 1725454042.765797 stat("/opt/cool/child-roots/873022-20fbac79/KU7MORiIpumls3Mj/lo", 0x7ffccb26be68) = -1 ENOENT (No such file or directory)
[pid 873027] 1725454042.765960 stat("/opt/cool/child-roots/873022-20fbac79/KU7MORiIpumls3Mj", {st_mode=S_IFDIR|0700, st_size=4096, ...}) = 0
[pid 873027] 1725454042.766126 stat("/opt/cool/child-roots/873022-20fbac79/BgfQpsFZYYNfGSyX/tmp/kit-crashed", 0x7ffccb26c4a8) = -1 ENOENT (No such file or directory)
[pid 873027] 1725454042.766289 stat("/opt/cool/child-roots/873022-20fbac79/BgfQpsFZYYNfGSyX/lo", 0x7ffccb26be68) = -1 ENOENT (No such file or directory)
[pid 873027] 1725454042.766450 stat("/opt/cool/child-roots/873022-20fbac79/BgfQpsFZYYNfGSyX", {st_mode=S_IFDIR|0700, st_size=4096, ...}) = 0
[pid 873027] 1725454042.766633 stat("/opt/cool/child-roots/873022-20fbac79/ORnuzrj6prc8Ei9P/tmp/kit-crashed", 0x7ffccb26c4a8) = -1 ENOENT (No such file or directory)
[pid 873027] 1725454042.766821 stat("/opt/cool/child-roots/873022-20fbac79/ORnuzrj6prc8Ei9P/lo", 0x7ffccb26be68) = -1 ENOENT (No such file or directory)
[pid 873027] 1725454042.766996 stat("/opt/cool/child-roots/873022-20fbac79/ORnuzrj6prc8Ei9P", {st_mode=S_IFDIR|0700, st_size=4096, ...}) = 0
[pid 873027] 1725454042.767188 stat("/opt/cool/child-roots/873022-20fbac79/gg9lFWFJbt8ZauWK/tmp/kit-crashed", 0x7ffccb26c4a8) = -1 ENOENT (No such file or directory)
[pid 873027] 1725454042.767375 stat("/opt/cool/child-roots/873022-20fbac79/gg9lFWFJbt8ZauWK/lo", 0x7ffccb26be68) = -1 ENOENT (No such file or directory)
[pid 873027] 1725454042.767556 stat("/opt/cool/child-roots/873022-20fbac79/gg9lFWFJbt8ZauWK", {st_mode=S_IFDIR|0700, st_size=4096, ...}) = 0
[pid 873027] 1725454042.767743 stat("/opt/cool/child-roots/873022-20fbac79/uVWT42GaSOTUuaEw/tmp/kit-crashed", 0x7ffccb26c4a8) = -1 ENOENT (No such file or directory)
[pid 873027] 1725454042.767948 stat("/opt/cool/child-roots/873022-20fbac79/uVWT42GaSOTUuaEw/lo", 0x7ffccb26be68) = -1 ENOENT (No such file or directory)
[pid 873027] 1725454042.768145 stat("/opt/cool/child-roots/873022-20fbac79/uVWT42GaSOTUuaEw", {st_mode=S_IFDIR|0700, st_size=4096, ...}) = 0
[pid 873027] 1725454042.768334 stat("/opt/cool/child-roots/873022-20fbac79/nDj3qUW5r9sZ7awK/tmp/kit-crashed", 0x7ffccb26c4a8) = -1 ENOENT (No such file or directory)
[pid 873027] 1725454042.768514 stat("/opt/cool/child-roots/873022-20fbac79/nDj3qUW5r9sZ7awK/lo", 0x7ffccb26be68) = -1 ENOENT (No such file or directory)

To take an example:

ls -ld QJ*
drwx------ 2 cool cool 4096 Aug 30 08:18 QJ90EiwQaou8PdUX

So - that should be cleaned; it contains:

ls -alR QJ90EiwQaou8PdUX/
QJ90EiwQaou8PdUX/:
total 40
drwx------ 2 cool cool 4096 Aug 30 08:18 .
drwxr-xr-x 745 cool cool 32768 Sep 4 12:44 ..

Nothing =)

[mmeeks]

So the check happens every 5 seconds in the forkit loop - surprising to see it as significant on the profile; but I guess ... 12 million stats per day for the 700 files is something (if I count right).

Not a hyper-high-priority, but would be nice to get cleaned up.