Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avoid host crash due to docker excessive resources use #193

Open
fe51 opened this issue May 15, 2024 · 0 comments
Open

Avoid host crash due to docker excessive resources use #193

fe51 opened this issue May 15, 2024 · 0 comments
Labels
help wanted Extra attention is needed question Further information is requested

Comments

@fe51
Copy link
Member

fe51 commented May 15, 2024

According to docker documentation :

By default, a container has no resource constraints and can use as much of a given resource as the host's kernel scheduler allows.

and this is how pyro-engine actually works.

However, we have recently encounter issues, rpi crash, probably due to a due to excessive use of multiprocessing and it has needed manual reboot.

from multiprocessing import Process, Queue, cpu_count

To prevent this kind of situation from recurring, we could consider limiting usage by docker Memory, CPUs, and GPUs.

I'd like to know the best practices on this subject (which parameters to play with ... hard or soft limits for memory..... if the rpi has 4 CPUs, is relevant to setup a limit to 3.8 .. etc ), which limitations to implement, and whether there are any constraints to implementing this type of docker limitation.

I would tend to think that we want to minimize the limits imposed on the docker service, but make sure we avoid a complete host crash

Thank you in advance for your suggestions.

@fe51 fe51 added help wanted Extra attention is needed question Further information is requested labels May 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted Extra attention is needed question Further information is requested
Projects
None yet
Development

No branches or pull requests

1 participant