Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SIGBUS errors (nnUNet PR) #438

Open
Kenneth-Schroeder opened this issue Mar 4, 2025 · 1 comment
Open

SIGBUS errors (nnUNet PR) #438

Kenneth-Schroeder opened this issue Mar 4, 2025 · 1 comment

Comments

@Kenneth-Schroeder
Copy link

FYI:
I have encountered several fatal SIGBUS errors using TotalSegmentator which uses nnUnetV2 under the hood.
The errors occurred when running TotalSegmentator on large inputs.
After some digging, I was able to boil the issue down to the predict_from_data_iterator function of nnUNet, which passes the (in my case quite large) prediction result tensors to worker processes. For this step, the tensors need to be serialized, which apparently can cause memory alignment issues and trigger SIGBUS errors.
I noticed the error would disappear if I removed the multiprocessing logic, but numpy arrays seem much more stable regarding serialization and fixed the SIGBUS errors even with multiprocessing, hence this nnUNet PR.

@wasserth
Copy link
Owner

wasserth commented Mar 5, 2025

Thank you for finding this bug and creating a PR with the solution. This is great!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants