Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Incorrect limits in dockerfile when cgroup v2 is used #106

Closed
funbiscuit opened this issue Sep 12, 2023 · 1 comment · Fixed by #107
Closed

Incorrect limits in dockerfile when cgroup v2 is used #106

funbiscuit opened this issue Sep 12, 2023 · 1 comment · Fixed by #107
Assignees
Labels
bug Something isn't working
Milestone

Comments

@funbiscuit
Copy link
Contributor

Currently script start.sh contains following code to automatically set memory limits:

if [ "$MEM_LIMIT_MB" = "" ]; then
	DOCKER_LIM_FILE="/sys/fs/cgroup/memory/memory.limit_in_bytes"

	if [ -e "${DOCKER_LIM_FILE}" ]; then
		MEM_LIMIT_MB=$(($(cat ${DOCKER_LIM_FILE})/1024/1024))
		echo "Using process mem limit of ${MEM_LIMIT_MB}MiB from ${DOCKER_LIM_FILE}"
	else
    	MEM_LIMIT_MB="1536"
    	echo "No process mem limit provided or found, defaulting to ${MEM_LIMIT_MB}MiB"
	fi
fi

But it doesn't work in my case in kubernetes, printing following message:

No process mem limit provided or found, defaulting to 1536MiB
MEM_LIMIT_MB=1536
Using default heap size of MIN(41% of MEM_LIMIT_MB, 640MiB) = 629MiB
HEAP_SIZE_MB=629

But there is a limit on this container (set by default by modelmesh controller) equal to 448Mi. This leads to OOMKilled because Java heap tries to grow to 629MiB which is greater than actual limit. I checked and actual limit is present at this path:

[app@modelmesh-serving-test-server-745dc7fb57-fxvxv mmesh]$ cat /sys/fs/cgroup/memory.max 
469762048

This is due to cgroup v2 that stores limits differently (at different path). Is it possible to change start.sh script so it works with both cgroup versions?

@ckadner
Copy link
Member

ckadner commented Sep 29, 2023

Thanks @funbiscuit

@ckadner ckadner added this to the v0.11.1 milestone Oct 2, 2023
@ckadner ckadner linked a pull request Oct 2, 2023 that will close this issue
@ckadner ckadner added the bug Something isn't working label Oct 2, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants