Update pod status contstants to account for Evicted and ContainerStatusUnknown and leverage them in sanitize

[j2udev]

I had a cluster that was blowing up with pods that were being Evicted and others that had ContainerStatusUnknown... figured I would try the new sanitize function, but noticed that it wasn't clearing them out. Updated accordingly.

I did notice some weird behavior in that the sanitize function didn't completely clear out everything in one run... (I had roughly 800-1000 failing pods 😱). I had to run it about 3 times to fully get everything. Is that expected?

I've said it before, but this tool is such a game changer. Thanks very much for all the hard work!

[derailed]

@j2udev Thank you for you kind words Joshua!
I haven't tried sanitize with that many pods. I think I had ~200 or so in a bad way.
Curious, do you use PriorityClass on your cluster. Wondering if we either get throttled or pod got rescheduled while sanitizing.
Currently we just log failed deletions but going to just stop sanitization if errors occur.
Would be great see the debug logs in the meantime and see if we errored out that would cause the sanitization to stop while in flight. Thank you for the details here!

[j2udev]

@j2udev Thank you for you kind words Joshua! I haven't tried sanitize with that many pods. I think I had ~200 or so in a bad way. Curious, do you use PriorityClass on your cluster. Wondering if we either get throttled or pod got rescheduled while sanitizing. Currently we just log failed deletions but going to just stop sanitization if errors occur. Would be great see the debug logs in the meantime and see if we errored out that would cause the sanitization to stop while in flight. Thank you for the details here!

We do use PriorityClasses, but not heavily. Unfortunately (fortunately?), our cluster has been healthy since then and it's not overly easy to test getting it back into that state... nor do I think my colleagues would appreciate me intentionally trying to put the cluster into that state 😅. If I see it pop up again though, I can grab the debug logs.

[derailed]

Lol - Right don't break your cluster on my account. Nw - will keep this in the pipe.
If you or other folks run into this issue. Please add the gory details. Thank you!!

[derailed]

@j2udev need TLC

[j2udev]

Sorry about that got busy with my day job

[derailed]

@j2udev Thanks for the updates Joshua!