RequestRateTooLarge in AppInsights

[Mortana89]

We're doing some load testing on our staging environment, and I'm seeing alot of the following in AI:
Microsoft.Azure.Documents.RequestRateTooLargeException

Cosmonaut.Diagnostics.CosmosEventCall+d__8`1.MoveNext

Shouldn't these type of errors be silent?

[Elfocrash]

Hello @Mortana89,

No I don't think that these errors should be log-level silent (even though they are app level silent) because it is very handy to aggregate them and see how many of your requests failed in order to generate some interesting metrics, autoscaling azure functions etc. You can also base alerts on them to indicate that you need to take immediate action in terms of scaling. I think it's a handy thing to log.

[Mortana89]

Hmm, I understand where you're going to, but the fact it's being logged under 'Exceptions' in AI is a bit annoying.
I expect (near to) zero exceptions in this view, because else something has a bug in it. When the system is under load, the fact that we receive 'RequestRateTooLarge' is totally normal behavior, which clutters the AI views now. It would be better (I think) if this could be moved to a dependency 'failure' then as it's not a real bug, merely a behavior of the system (Cosmos DB) under pressure. What do you think?

[Elfocrash]

Exceptions are not meant to indicate bugs but rather exceptional behavior and I think this specific one falls under this category. I could potentially add a flag on the AI package level to eagerly ignore it but I know for a fact that many users base alerts and autoscaling on it (including myself) and I can't just remove it.

[Mortana89]

Why not use the Cosmos DB monitoring for that? That's what we do, we send alerts when the cosmos db throttled requests are abnormally high.
A flag would be useful, would that be much work?

[Elfocrash]

Nah it shouldn't be much work. The reason for the code level logging is consistency of logs. It leads to simpler and more consistent queries.