Guidance for system and process metric instrumentation using a Collector

[braydonk]

Area(s)

area:system

What's missing?

The system and process namespace metrics are generally reflections of values that any process can conceivably collect from the operating system. However, there are two caveats with metrics from these namespaces:

• Forsystem metrics, it will rarely if ever make sense for multiple sources on the same machine to report the metrics
• process metrics can be relatively expensive to collect, so large numbers of processes instrumenting themselves can lead to unwanted overhead

Describe the solution you'd like

We should find a way to write some manner of guidance around the collection of these metrics. Many of them have relied on our group being inherently tied up with the hostmetricsreceiver in the Collector, and so this alternate angle has not been considered. However there are self-instrumenting process libraries and it is surely within the realm of possibility that alternate implementations than the hostmetricsreceiver to collect system information could work fine. So we should think more holistically about how users may interface with these conventions outside of a Collector, and recommend against bad approaches.

This guidance should exist in the Non-Normative section for the System group (in docs/non-normative/groups/system).