APM is processing tons of metric events

[SimonSimCity]

Using my monitoring tool for the APM of my ELK stack, I see that it processes a ton of metric events. When restarting the meteor application, it drops to almost 1k/minute, but it keeps steadily rising again - slowly but steadily, and it looks it rises the more transactions or spans are reported by the system. I got it skyrocketing up to 80k/m. The amount of incoming requests per minute is constantly around 45/m, so I guess it has to do with the data that's sent per request.

Any hint?

[kschingiz]

@SimonSimCity yes, we have found performance issues caused by meteor-elastic-apm. We are currently investigating it. I think this issue is related to meteor-measured package. These issues appeared when I have integrated meteor-elastic-apm with meteor-measured.
Tomorrow I solve it.
Try to disable meteor-measured and let me know the result.

[SimonSimCity]

I'll only disable the additional metrics added in this package, just to see if this problem is related to this package or to elastic-apm-node. Looking forward to your result.

[SimonSimCity]

Maybe an update to version 2.15.0 of https://github.com/elastic/apm-agent-nodejs would already help it: elastic/apm-agent-nodejs#1273

I could at least prove that it still increases when disabling the metrics added by this extension.

[kschingiz]

@SimonSimCity seems like it's issue from elastic nodejs agent. I will try to update the version. Thanks!

[SimonSimCity]

I've now a bit of time to report my findings. The update to 2.15.0 in deed reduced the amount of metrics sent to the APM system, but on the cost of harming other metrics.

I now e.g. do not get a metric for my event-loop-delay anymore if there's a high load on the server. I'll have to debug this and come up with a proper approach to tackle it.