Skip to content
This repository has been archived by the owner on Sep 25, 2023. It is now read-only.

[BUG] Linux OOM killer reaps cuSignal process on a Jetson Xavier NX #341

Open
znmeb opened this issue Mar 23, 2021 · 4 comments
Open

[BUG] Linux OOM killer reaps cuSignal process on a Jetson Xavier NX #341

znmeb opened this issue Mar 23, 2021 · 4 comments
Labels
? - Needs Triage Need team to review and classify bug Something isn't working inactive-30d inactive-90d

Comments

@znmeb
Copy link

znmeb commented Mar 23, 2021

Describe the bug
When running the E2E_Example notebook with cuSignal 0.18.0 on a Jetson Xavier NX (8 GB of RAM), JupyterLab reports a
kernel restart running "Run Periodogram with Flattop Filter over Each Row of Ensemble" on the GPU.

Steps/Code to reproduce bug

  1. Install cuSignal in a Conda environment as described in the documentation
  2. Install JupyterLab in the same environment
  3. Start JupyterLab, browse to the notebooks and open E2E_Example.
  4. Run the cells one at a time from the top down until it crashes.

Expected behavior
All cells run

Environment details (please complete the following information):

Additional context
Linux behaves gracelessly if you ask it for more RAM than it has available. It will either thrash / catastrophic swap, rendering the system unresponsive, or kill processes of its own choosing. Neither will allow you to progress.

In this case it appears it chose the second option - here's a journalctl log:

journal.txt

This isn't mission-critical for me; I can page through the notebooks and find cases that will run. I doubt if I'll run into something
like this in my application code.

@znmeb znmeb added ? - Needs Triage Need team to review and classify bug Something isn't working labels Mar 23, 2021
@awthomp
Copy link
Member

awthomp commented Mar 23, 2021

Hey @znmeb -- Does the notebook work if you just reduce the size of the signal ensemble generated on GPU at the beginning of the notebook?

@znmeb
Copy link
Author

znmeb commented Mar 23, 2021

I haven't tried that yet - I did that with some other notebooks a few months ago and it worked, so my guess is it will work. Which variable controls the size?

@github-actions
Copy link

This issue has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be labeled inactive-90d if there is no activity in the next 60 days.

@github-actions
Copy link

This issue has been labeled inactive-90d due to no recent activity in the past 90 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
? - Needs Triage Need team to review and classify bug Something isn't working inactive-30d inactive-90d
Projects
Status: No status
Development

No branches or pull requests

2 participants