Drastically improve zombie process cleanup efficiency #362
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Address an issue where using `xargs` caused argument lists to become too long on systems with many processes. This resulted in errors such as:
/usr/local/bin/kill_zombie_parents.sh: line 14: /usr/bin/ps: Argument list too long
Changes include:
- Replacing `xargs` with shell loops for reading process IDs in `detect_zombies` and `find_zombie_parents`.
- Using `while read` loops to safely build the list of child processes without exceeding argument length limits.
- Ensuring more robust zombie process detection and signaling by avoiding argument overflows.
This update ensures the script works reliably even in environments with a large number of processes.
Affected functions:
- Modified zombie process handling logic in the `detect_zombies`, `pidtree`, and `find_zombie_parents` functions.
This commit overhauls the zombie process cleanup script, addressing a critical performance issue: - Previously, the script would scan all processes and zombies, gathering parent PIDs for EACH zombie. This approach was extremely inefficient, taking up to 20 minutes when the server had 465,276 zombie processes. - Now, the script stops after finding the first zombie process and focuses on terminating its parent. This approach is based on the understanding that multiple zombies are typically linked to a single parent process. Terminating the parent process will make all of its zombie children disappear, drastically improving efficiency. Other improvements include: - Add safety checks to avoid terminating containerd-shim processes - Remove the zombie threshold check, allowing immediate action on all zombies. This prevents the misconception that a small number of zombie processes is acceptable and encourages users to properly address the root cause by integrating process reapers like tini. - Increase crontab execution frequency from every 15 minutes to every 5 minutes These changes result in a much more responsive and efficient handling of zombie processes, reducing cleanup time from 20 minutes to just seconds, even with hundreds of thousands of zombies.
Test results
Have additionally tested on a production server with |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This commit overhauls the zombie process cleanup script, addressing a critical performance issue:
Previously, the script would scan all processes and zombies, gathering parent PIDs for EACH zombie.
This approach was extremely inefficient, taking up to 20 minutes when the server had 465,276 zombie processes.
Now, the script stops after finding the first zombie process and focuses on terminating its parent.
This approach is based on the understanding that multiple zombies are typically linked to a single parent process.
Terminating the parent process will make all of its zombie children disappear, drastically improving efficiency.
Other improvements include:
that a small number of zombie processes is acceptable and encourages users to properly address the root cause
by integrating process reapers like tini.
These changes result in a much more responsive and efficient handling of zombie processes,
reducing cleanup time from 20 minutes to just seconds, even with hundreds of thousands of zombies.