unpackerr consuming insanely high CPU and IO

[SteffanCline]

On Ubuntu 22 64bit I was running unpackerr_0.11.2-505_amd64.deb and saw in the logs that there was an error about not being able to work with the log file even after checking permissions and ownership. Realizing that was an old version, I archived the log file and upgraded to unpackerr_0.14.5-797_amd64.deb. It immediately started ramping up the CPU to 200-400% and was consuming all the IO until it pulled a beefy server to it's knees. I checked iotop and shortly after restarting unpackerr I saw that it had read over 80G in a very short time. Stopping unpackerr allowed my server to go back to normal. Any ideas what would cause this to happen?

[SteffanCline]

Reopened issue. I only have 1 parallel process set and unpackerr keeps ramping up ridiculously high unlike the older version or when I ran this on a mac. I have to use cpulimit to limit the processor otherwise every few minutes unpackerr locks up the server for a few minutes at a time. There are active downloads in transmission however there zero just completed downloads that would explain this.

[davidnewhall]

Please provide the log file.

[SteffanCline]

I forgot to turn on the switch for logging in the new config I made with your configurator so everything dumped to syslog. I extracted that and have attached it. I redid the config and turned on logging into it's own file. I can provide a new sample of that later.

[davidnewhall]

Removing things from queue will likely alleviate the problem. This is a bit much.

02-02T09:34:00.562864-07:00 localhost unpackerr[2385360]: [INFO] 2025/02/02 09:34:00 [Radarr] Updated (http://127.0.0.1/radarr): 96 Items Queued, 96 Retrieved
2025-02-02T09:34:00.635148-07:00 localhost unpackerr[2385360]: [INFO] 2025/02/02 09:34:00 [Sonarr] Updated (http://127.0.0.1/sonarr): 956 Items Queued, 956 Retrieved
2025-02-02T09:34:03.590995-07:00 localhost unpackerr[2385360]: [INFO] 2025/02/02 09:34:03 [Lidarr] Updated (http://127.0.0.1/lidarr): 380 Items Queued, 380 Retrieved

[SteffanCline]

I have no idea how those can be in the queue. I have items downloading SLOW from the lack of seeders however nothing is done and decompressing.

[davidnewhall]

I'm not sure I understand what you mean. Go into Sonarr, click Activity -> Queue. Remove 900+ things.

[SteffanCline]

I see items in that list I know i have downloaded from elsewhere because of failures etc. If I do that, it says it will remove them from the download client. Won't that destroy my rankings on the torrent trackers?

[davidnewhall]

Sounds like you forgot to setup categories for your downloads. See questions 2 and 3. https://unpackerr.zip/docs/unpackerr/faq

[SteffanCline]

I see what you're saying about the Activity. I removed all the NZBGet because they failed and I replaced them from other sources. I wish it would remove those when a scan shows they are complete. Anyway, I'll do some reading but immediately, I don't see a way to configure categories. AFAICT, that's set by Sonarr, Radarr, Lidarr etc.

[davidnewhall]

I don't see a way to configure categories.

https://trash-guides.info/Downloaders/NZBGet/Paths-and-Categories/

[SteffanCline]

I do have categories already configured in NZBGet but I don't have the specific paths for each category set, they use DestDir, because the *arrs set the download path. I don't have unpackerr set to monitor nzbs, only torrents.

[davidnewhall]

You have 900 things in Sonarr queue. That's the problem. Those things get into queue based on the category you've chosen. You need to reduce the queue to less than 100 for things to be less chaotic.

[SteffanCline]

I will look those over. Some are definitely wishlist items missing from a collection. If I were to disconnect unpackerr from sonarr, wouldn't unpackerr find those downloads and decompress them then sonarr scanning for completed downloads pick up the unpackerred items?

[davidnewhall]

Sorry I don't understand your question. if you remove the sonar config from unpackerr it wont extract any of the things in Sonarr queue. How many of the 900 items are you expecting unpackerr to extract? Why are there incomplete "wishlist" items in queue? This is not making sense and I think you're using Sonarr incorrectly. 900 things in queue is the symptom of incorrect use/configuration, usually category related.

[SteffanCline]

Some of the 900 were rare items that have been downloading then the seeders come on and go offline every so often and it takes a long time for those to finish. I can't just remove those from transmission otherwise I lose credit for seeding what I have. I have cleared out as many of those as I can. Sonarr/Radarr are well under 100. We'll see how it goes. Odd that the older version didn't have this problem.

[davidnewhall]

I wasn't recommending removing them from Transmission, just Starr app queues.

[SteffanCline]

I understood that but when removing them in the activity section, it appears to remove them from transmission too. Notice the dropdown is disabled.

[davidnewhall]

Change the category on the item in Transmission to remove it from the queue in a Starr app.

[SteffanCline]

That's one of the problems I've seen with Transmission, there is no category setting. It just has the download path. What you're saying works perfect with NZBGet.

[davidnewhall]

Transmission on macOS calls them groups. I believe Transmission on Linux calls them labels.

[SteffanCline]

Thanks, found the labels on Transmission. I'll have to see if there's a way that the *arrs can set these in Transmission. Even with the Activity list way down, it's still going crazy on CPU.

[davidnewhall]

What is it doing that's making it go crazy? strace it? is there disk io? Need details and a log file.

[davidnewhall]

Still seems to be the same problem.

[INFO] 2025/02/02 11:25:47 [Radarr] Updated (http://127.0.0.1/radarr): 97 Items Queued, 97 Retrieved
[INFO] 2025/02/02 11:25:48 [Sonarr] Updated (http://127.0.0.1/sonarr): 786 Items Queued, 786 Retrieved
[INFO] 2025/02/02 11:25:50 [Lidarr] Updated (http://127.0.0.1/lidarr): 380 Items Queued, 380 Retrieved

[davidnewhall]

All of these things that say Completed item still waiting need to be removed from queue.

[SteffanCline]

Current counts- Sonarr: 20, Radarr: 13, Lidarr: 367. I'm working to reduce Lidarr right now. So, those counts in the logs are off. They've been updated now for about 20 min. Should I clear the log file and restart unpackerr?

[davidnewhall]

So, those counts in the logs are off.

Please see the FAQ, question 2.

[SteffanCline]

Unknown is checked in all three *arrs. I did that earlier when you told me.

[davidnewhall]

The log you sent ends at 11:25. Send me the current one.

[SteffanCline]

That was the most current. I had just downloaded it. Let me clear it out and restart unpackerr.

[davidnewhall]

Perhaps tail the log to figure out if it's freezing and if that's the point at which memory pressure goes up.

[SteffanCline]

Interesting. I see this in syslog. That's what I had before. I don't know what's generating that. I don't have any log rotation turned on.

2025-02-02T14:04:06.631961-07:00 localhost unpackerr[2743732]: Logger Error: error rotatorring: error renaming log: rename /var/log/unpackerr.log /var/log/unpackerr-2025-02-02T14-04-06.629.log: permission denied
2025-02-02T14:04:06.632070-07:00 localhost unpackerr[2743732]: Logger Error: error rotatorring: error renaming log: rename /var/log/unpackerr.log /var/log/unpackerr-2025-02-02T14-04-06.630.log: permission denied

[davidnewhall]

It's on by default, so you probably do have it turned on. Give the app write permission to the folder you put the log file in. Recommend /var/log/unpackerr/unpackerr.log over /var/log/unpackerr.log

[SteffanCline]

Updated unpackerr.conf to use the log path you provided. I cleared the log file and restarted unpackerr. Now the counts are right:

[INFO] 2025/02/02 14:10:47 [Radarr] Updated (http://127.0.0.1/radarr): 13 Items Queued, 13 Retrieved
[INFO] 2025/02/02 14:10:47 [Sonarr] Updated (http://127.0.0.1/sonarr): 20 Items Queued, 20 Retrieved
[INFO] 2025/02/02 14:10:48 [Lidarr] Updated (http://127.0.0.1/lidarr): 304 Items Queued, 304 Retrieved

[SteffanCline]

While I do see it writing to /var/log/unpackerr/unpackerr.log, I still see things being written to /var/log/syslog. Is that normal?

2025-02-02T14:14:49.178426-07:00 localhost unpackerr[2792770]: [INFO] 2025/02/02 14:14:49 [Unpackerr] Queue: 121 waiting, 0 queued, 0 extracting, 1 extracted, 0 imported, 0 failed, 0 deleted
2025-02-02T14:14:49.178495-07:00 localhost unpackerr[2792770]: [INFO] 2025/02/02 14:14:49 [Unpackerr] Totals: 0 retries, 0 finished, 0|0 webhooks, 0|0 cmdhooks, stacks; event:0, hook:0, del:0, up 4 mins 14 ms 762 µs

[davidnewhall]

Yes. Turn it off by setting quiet to true. https://unpackerr.zip/docs/install/configuration#global-settings

[SteffanCline]

Thanks. I didn't realize stderr/stout was syslog but it makes sense. So far, CPU is down without cpulimit so that's good. I'm watching a CPU intensive move in Plex so we'll see how it does. Thanks for the help!

[SteffanCline]

So far, so good. The number of files and config did not change between unpackerr_0.11.2-505_amd64.deb and unpackerr_0.14.5-797_amd64.deb. What explains why it was fine with old and not new? Just curious.

[davidnewhall]

The misconfiguration with the log file was probably causing this. When I find time I'll try to reproduce it.