Missed attestations, beacon node increase CPU, increased disk read + writes #5105
Labels
optimization
Something to make Lighthouse run more efficiently.
tree-states
Upcoming state and database overhaul
Comments
|
Could be related to this issue, which will be fixed in v4.6.0: #4918 Please try upgrading once the release is available (soon) and let us know if it's resolved |
|
Will be happy to do so |
|
I haven't upgraded my node, but it's been 7 days since CPU has been higher than 75.0 on average during a 10m period. This is CPU over the last 30 days. Each of the spikes that took place before Jan 23 in the chart below had triggered the behavior explained in this ticket. We haven't seen this behavior since Jan 23. 
We will update to the latest lighhouse release and comment. |
|
Spoke to soon. Just had another episode a few hours ago. Upgrading to the latest now. 
|
|
|
The incidents had significantly reduced in frequency, but we still see them in our node. However, today they've come back stronger than never before! These are the charts from Sunday Feb 11 from 17:00 to 18:00 UTC 
Looking at the logs, I see the process is getting killed by systemd. We're having a second, nastier, incident just now (Feb 13, 17:14 UTC) which is ongoing. 
Logs show also a restart: The server continues to restart: |
It got oom-ed (out of memory error) so it was forced shut down and restarted (set in systemd) |
No, there is a single validator connected to the beacon node and nothing else doing API queries to the beacon node. |
|
@vogelito Could you please DM me debug logs and metrics from one of these node on Discord (I am |
|
Sent! |
|
Hi - we’re seeing another big instance of this happening today after quietness for a few days |
|
It's been pretty bad over the last 48 hours. 
Sending logs via discord. |
|
Started running the new version of lighthouse on Just over 2 hours after upgrading ( Similar CPU spike to past times: 
I increased RAM to 32GB from 16GB to see if that helps...: |
|
Since increasing the memory of the node to 32GB (on |
|
Sounds good! Let’s see how it goes after the new release… |
|
Some more logs prior to our memory upgrade worth sharing: This is what the last event looks like: |
|
@vogelito I think #5270 should help quite a bit. Lighthouse is not using particularly excessive amounts of memory (which would be indicative of a bug). All your OOM logs show it gettting killed at around 8GB:
( On our infrastructure Lighthouse uses 4-8GB regularly, so I expect the spikes to 8GB were too much for your node with the 16GB hard limit. Geth is using the rest of the memory by the looks of things, its RSS shown in the kernel backtrace is |
|
Sounds like a plan to me.... |
|
Upgraded to
Will report back! |
|
It's already been killed: Seems like Geth was using ~6.38GB: 
Let me know if you need anything else on my end to be helpful in your debugging. |
|
Another crash: |
|
@vogelito Please go back to 32GB or try |
|
If you can DM me debug logs again too, we can take a look. Lighthouse shouldn't really be hitting 9GB. |
|
Looking progressively worse.... I've restarted with DM'ed you logs on Discord |
|
The issue continues after the restart, though at a lesser frequency |
|
@vogelito Can you please DM me a dump of We don't have any imminent memory improvements coming, so Lighthouse's memory usage is expected to remain spikey until we merge |
|
Here's
|
|
tree-states looks promising :) I've sent you the output of log of restarts, still running 16GB, will restart to 32GB now until |
|
I updated to the Tree People release last Thursday (June 13). I see a smaller and less choppy memory footprint but with a clear upwards trend. See chart below: Do we want to give 16GB RAM a try? |
|
@vogelito It should plateau. It won't grow indefinitely |
|
Try the 16GB RAM, I think it should be OK now |
|
Alright, back to 16GB RAM. Will report any findings.... |
|
No restarts in the last 28 hours running 5.2.0 on 16GB RAM. Will report again in a week. |
|
I can confirm that everything look very nice and stable on a 16GB machine for the first time in 7 months :) Thanks for the hard work!! Let me know if you need anything from me or if I can be helpful in any way! |
|
Great to hear @vogelito! Do you mind if we close this issue? |
|
Of course! Well earned close!! |
|
Closing the issue as discussed above |
|
Although definitely no longer an issue, wanted to give you visibility over a couple of OOM kills that have taken place since I downgraded to a 16GB machine June 30th 17:24:00 until 18:24:00 July 02 19:30:00 unti 20:30:00 |
|
As always, happy to help debug. |
|
Hi, The OOM kills are back with a vengeance for the last 2 days. Nothing changed in our setup. Same behavior as before.... |
|
It's been running 5.3.0 since Aug 14, but kills just took place over the weekend: |
How much memory is on the node again, was it 16GB or 32GB? If it is 16GB, it is recommended to upgrade to 32GB Which execution client are you using? |
|
Hi @vogelito. Looking at our own nodes I'm also seeing some occasional memory spikes around 7-8GB. I'm investigating now. It may not be an easy fix however, so CK's suggestions of 32GB is probably the way to go in the short term. |
michaelsproul
added
optimization
|
Using geth and running on 16GB. It had run without issues since mid August but I’ll go back to 32GB for now. happy, as always, to help debug if I can be helpful! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description
5 weeks ago I started noticing my validator node missing attestations. I then realized these missed attestations matched moments where CPU usage in my node was high. I then added a probe to see what process in my node was spiking the CPU usage and realized it was mostly due to higher CPU usage from my beacon node. I also realized these periods matched periods of higher than usual disk reading and writing.
This is new behavior that started roughly 5 weeks ago.
Version
I'm downloading the release binaries from Github and currently running:
Present Behaviour
I have missed attestations on each of these periods of high CPU usage:
Expected Behaviour
I shouldn't be missing attestations.
Steps to resolve
I'm unaware of what steps I can take to resolve this, but happy to work with the devs on resolving them :)
The text was updated successfully, but these errors were encountered: