-
Notifications
You must be signed in to change notification settings - Fork 27
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
omnitrace hangs before hostCallback function #307
Comments
Could you provide a backtrace? There is usually one printed out when you hit Ctrl+C. I suspect there is something funny going on in roctracer, which delivers callbacks to omnitrace about the HIP calls. Can you try disabling roctracer support and see if it still hangs? Could you also try running it with rocprof and seeing if it still hangs? |
Below is the full output of the program, after hitting ctrl+c. Unfortunately I don't have time to investigate the other things right now, will get back to it on Monday.
|
So I tried running it with rocprof, and it also hangs, it seems that in the first
If I comment out the Running the program with |
Yes, it seems like a fundamental issue in roctracer (i.e. outside of the scope of omnitrace). I’ll pass on the bug report and see if it can get patched. |
Hello,
I am trying to trace my AMDGPU application with Omnitrace, but I am running into an issue with a host callback function. Using
hipStreamAddCallback
I submit a host function into a stream. Without Omnitrace, the program works as expected. But with Omnitrace, the program hangs and the host function is never launched.Reproducer program:
When running it without omnitrace, the program correctly outputs
but with omnitrace, it only outputs
and then nothing, then it just hangs, seemingly forever.
I am compiling the program using
And runing using
omnitrace-instrument
seems to have the same problem.I am on LUMI-G compute node (MI250x), using rocm-5.2.3 (the only one properly supported there,
module load LUMI/23.03 rocm/5.2.3
).I installed omnitrace using this guide, just running the installation script and adding the appropriate directories to
PATH
andLD_LIBRARY_PATH
.In case I missed any details, please ask.
I would appreciate any help.
The text was updated successfully, but these errors were encountered: