Attribute injection in the Collector

[jade-guiton-dd]

Problem

The Collector SIG is working on various improvements to the Collector's internal telemetry, among which is an RFC specifying attributes used to identify which Collector component emitted a given item of telemetry (eg. "the transform/foo processor in pipeline metrics/bar").

We are looking for ways to attach these "component attributes" to internal Collector telemetry automatically, without needing component developers to manually add them. (See the discussion in this issue.)

Current solutions

This has proven to be complicated with the current OTel API and data model:

• resource attributes are set once when creating a Provider, and cannot be changed without recreating the telemetry exporting pipeline, whereas we need different values for different components;
• instrumentation scope attributes have the wrong semantics (they identify parts of the code, not runtime component instances), and are poorly supported (if at all) by observability backends;
• span/metric/log attributes are normally set in the final call, inside component code which we don't control.

Our current best solution (in this PR) uses span/metric/log attributes, and involves wrapping the TracerProvider and MeterProvider we pass to the component, as well as the Tracer, Meter, and all instruments created from them, to thread the attributes through the component and inject them in the final call. This works, but is brittle and inelegant.

Another proposed idea is to thread the attributes through the Context passed to components, and inject the attributes using SDK processors. However, this would require some cooperation on the component side (carefully threading Contexts and never using context.Background), and because metric processors do not exist, would still require an unwieldy wrapper-based workaround.

Question

Does the Go SIG know of any current or planned SDK-level or API-level features we could leverage to make this easier to accomplish?

Here are some ideas we thought of which would fit our use case:

• the ability to define default span/metric/log attributes on Tracers/Meters/Loggers
• the ability to specialize a Provider with additional resource attributes, without recreating the exporting pipeline
• the ability to define "metric processors", akin to span/log processors
  (note that there is an ongoing specification effort related to this)
  • perhaps additionally, the ability to define extra Baggage associated with a Meter/Tracer/Logger, to avoid issues with threading Contexts through components

If you have ideas for alternative solutions to this using purely the OTel API, we are of course also interested.

Additional context

Possibly related use cases:

• There are features in specific instrumentation libraries in go-contrib which allow adding arbitrary attributes to the emitted telemetry: in otelhttp, in otelgrpc, work in progress
• This feature request in the Collector could probably be solved with metric processors

[jmacd]

@jade-guiton-dd Thank you, this is great. I would like to propose to elevate this to the specification level, as I do not believe the stated problems are specific to the OTel Collector. You've listed three solutions and all of them have been discussed in the past and yet we are stalled.

Here are some brief positions of mine:

instrumentation scope attributes have the wrong semantics (they identify parts of the code, not runtime component instances), and are poorly supported (if at all) by observability backends;

I think this is an unfortunate situation, and I would focus our efforts on resolving this. IMO InstrumentationScope is the correct place for these attributes to go; while the name and version refer to a body of source code, the attributes field was originally intended to do what you're looking for. It's OpenTelemetry's problem that we've had a scope attributes field for years and still can't use it. Let's fix this.

I think this is the best approach compared with the others you mentioned. Our goal should be to avoid repetition of attributes for each span/metric/log because it is an important optimization.

span/metric/log attributes are normally set in the final call, inside component code which we don't control.

OpenTelemetry has (for years) talked about a specific solution in metric instrumentation called the "Bound" instrument, which is a bound instrument for metric-ing with pre-specified attributes. Many metrics and logging APIs outside OTel allow you to register attributes that are automatically inserted to every log or metric (e.g. zap's Logger.With(), Prometheus' MetricVec.CurryWith()), and still there's no way to do that in OpenTelemetry. While I think OTel should move in this direction, I would still prefer to see component-level attributes as stated in open-telemetry/opentelemetry-collector#12217 use instrumentation-scope attributes to avoid repetition.

Another proposed idea is to thread the attributes through the Context passed to components, and inject the attributes using SDK processors. However, this would require some cooperation on the component side (carefully threading Contexts and never using context.Background), and because metric processors do not exist, would still require an unwieldy wrapper-based workaround.

See the discussion in open-telemetry/oteps#207. Again, this is something I think OpenTelemetry should work out. You linked to the work on MetricsProcessor, which goes part of the way to a solution: OTEP 207 takes it further by proposing that OTel provide a standard way to enter context that would automatically be extracted by the SDKs. I would like to see such functionality in the Collector, but I would not use it for component-level attributes. The place I would use this in a Collector, for example, is to attach information propagated in-band that is not component-specific, like information about the end-user who submitted the request or the identity of the downstream destination calculated by an upstream load balancer. (Also yes, Go developers should always be careful about correct use of context.)

[mx-psi]

@jmacd

It's OpenTelemetry's problem that we've had a scope attributes field for years and still can't use it. Let's fix this.

It's unclear to me how we fix this. Do you have a concrete proposal on how to go about it?