Daily crashes of Frigate 0.14 because of NAS flushing memory daily

[aberlioz]

Bug description

Alright so this is potentially not a Frigate bug, let me explain.

I own a QNAP NAS, on which I have installed Frigate for two years now, using Container Station and a docker compose stack deployed through portainer. I'm passing through the devices correctly (including a coral USB), and computed the shm_size following the recommendation and everything was rock solid for a couple of years.

Last week, I decide to switch to Frigate 0.14, migration went smoothly and everything works as expected. But after that, every morning I wake with Frigate crashed (ie not reading camera feeds with the error No frames have been received, check error logs) with these error messages: Fatal Python error: Bus error.

Some research both on github and documentation quickly shows that it happens when the shm_size size is too low. I recheck my computation, still correct, I increase nonetheless just in case to 200mb (I have 4 camera detecting at 720p so it should be overkill). Unfortunately the issue still happens every morning.

I decide to check my monitoring and discover that the cached memory of my frigate docker container (and all other docker containers) is drastically reduced daily at 01:00!!

Looking at historical data, it seems this behavior has always been there, it's not something new. I searched the web without finding anything, so I ended up logging in my QNAP through ssh (something I rarely need to do) and start looking around. Logs did not bring anything, but remembering the timing of this issue I check the crontab and oh boy what do I find?! Among the 40 cron defined, I find one the suspect:

0 1 * * * /etc/init.d/flush_memory.sh >/dev/null 2>&1

This script does 2 things

/proc/sys/vm/drop_caches
/proc/sys/vm/compact_memory

This process seem to be called Qboost Memory Optimization, but there isn't much information about it.

Anyway to confirm my theory, I manually disable the script and wait another day. Guess what happens ? Frigate is running fine, no more cached memory drops, no more errors, issue is gone.

Questions

Alright now I know which process is responsible for this. But this behavior is not new on QTS side, so my question is:

Why do I start experiencing this issue with the new version of frigate ? Is there anything that can be done so that frigate can recover on its own ?

Seems like frigate could recover on its own from this in version 0.13.2. All my others docker containers are also "fine" with this so-called "memory optimization".

[deanpm70]

Not a fix but as a temporary workaround you could add your own cron jobs to stop and restart frigate either side of the qboost job.

[NickM-27]

Not sure this has anything to do with that other than the reduced CPU usage so frigate doesn't slow down too much. There are many posts in this repo of users having the issue, I don't think it's related to this. 200mb isn't really that much, I'd recommend trying 1024MB

[aberlioz]

I will update the shm_size to 1024MB and see what happens then.

[aberlioz]

Indeed @NickM-27 after updating the shm_size to 1024MB there is no issue anymore!!

First thank you for the fix, happy that my frigate is still running 🙂

But I was following the official documentation here

And even the example states:

# Example for eight cameras detecting at 1280x720, including logs
$ python -c 'print("{:.2f}MB".format(((1280 * 720 * 1.5 * 9 + 270480) / 1048576) * 8 + 30))'
126.99MB

So would you would happen to know why following these recommendation did not work for me ? Should we update the documentation ?

[NickM-27]

The docs are a recommended starting point, if your system is running slower or has load from other services it can slow things down.

And no update is needed because we've rewritten the way that SHM works for the next version