[middleware] turn off noisy RPC call observations

[acpana]

overview

A complex consul cluster will make various RPC calls for internal operations. For instance, Status.RaftStats or Coordinate.Update. Unfortunately, these calls do not actually show a lot of value and they can cause more noise in the metrics than offer actual signals.

proposal

We need to find a way to "filter out" RPC calls that we don't really want to be observed, for whatever reason. Ideally, this is a dynamic or "reloadable" operation, in that it's not going to require restarting all Consul server.

Let's look at a couple proposals.

P1

Label based filtering is supported by go-metrics. However, our consul.rpc.service.call metric looks like this:

{
"metric": consul.rpc.service.call,
"label": value
...
}

If we were to filter out a label, then all labels get filtered out. We only want to filter out the label when its value matches == Coordinate.Update (for example).

That is why we may want to implement hashicorp/go-metrics#131,

This proposal is the recommended way to solve this since it supports dynamic filtering reload.

P2

It may be that implementing P1 is too heavy, or, indeed, that implementing it in go-metrics does not make sense.

Then the layer for us to implement at would the RequestRecorder. We would proably need to add:

• config support for which label values we don't want to support
• add interceptor style to any go-metrics call to take out the filtered our label values (or just start with the RequestRecorder
• make said config hot reloadable