Support for http.request.header.<key> and http.response.header.<key> attributes

[Kielek]

Issue that does not fit into other categories

What are you trying to achieve?

Possibility to opt-in following parameters

• http.response.header.<key>,
• http.request.header.<key>.

Needed at last in

• Instrumentation.AspNet,
• Instrumentation.AspNetCore,
• Insrtumentation.Http,
• Instrumentation.GrpcNetClient

Potentially affects also Instrumentation.Owin in this repository.

It will be great to have a possibility to enable it by options and environmental variables.

Additional context.

Java already implements it. It can be configured by 4 independent env. variables:

• OTEL_INSTRUMENTATION_HTTP_CLIENT_CAPTURE_REQUEST_HEADERS
• OTEL_INSTRUMENTATION_HTTP_CLIENT_CAPTURE_RESPONSE_HEADERS
• OTEL_INSTRUMENTATION_HTTP_SERVER_CAPTURE_REQUEST_HEADERS
• OTEL_INSTRUMENTATION_HTTP_SERVER_CAPTURE_RESPONSE_HEADERS

Needed by (Splunk) .NET Auto Instrumentation technology. Would like to contribute it here.

[Kielek]

Implementation proposal

Changelog

Options class status

only collection found is public API files across main/contrib repositories IReadOnlyCollection<string> UriPrefixes.
There are also 2 instances of IReadOnlyDictionary<string, object> in Geneva exporter.

Extending public API changes

Option 1 - same naming - for server and client

Same API names, regardless if it is used on the server or client side

Changes for OpenTelemetry.Instrumentation.AspNet package:

+OpenTelemetry.Instrumentation.AspNet.AspNetTraceInstrumentationOptions.CaptureRequestsHeaders.get -> System.Collections.Generic.IReadOnlyCollection<string!>?
+OpenTelemetry.Instrumentation.AspNet.AspNetTraceInstrumentationOptions.CaptureRequestsHeaders.set -> void
+OpenTelemetry.Instrumentation.AspNet.AspNetTraceInstrumentationOptions.CaptureResponseHeaders.get -> System.Collections.Generic.IReadOnlyCollection<string!>?
+OpenTelemetry.Instrumentation.AspNet.AspNetTraceInstrumentationOptions.CaptureResponseHeaders.set -> void

Changes for OpenTelemetry.Instrumentation.AspNetCore package (if nullable enabled):

+OpenTelemetry.Instrumentation.AspNetCore.AspNetCoreTraceInstrumentationOptions.CaptureRequestsHeaders.get -> System.Collections.Generic.IReadOnlyCollection<string!>?
+OpenTelemetry.Instrumentation.AspNetCore.AspNetCoreTraceInstrumentationOptions.CaptureRequestsHeaders.set -> void
+OpenTelemetry.Instrumentation.AspNetCore.AspNetCoreTraceInstrumentationOptions.CaptureResponseHeaders.get -> System.Collections.Generic.IReadOnlyCollection<string!>?
+OpenTelemetry.Instrumentation.AspNetCore.AspNetCoreTraceInstrumentationOptions.CaptureResponseHeaders.set -> void

Changes for OpenTelemetry.Instrumentation.GrpcNetClient package (if nullable enabled):

+OpenTelemetry.Instrumentation.GrpcNetClient.GrpcClientTraceInstrumentationOptions.CaptureRequestsHeaders.get -> System.Collections.Generic.IReadOnlyCollection<string!>?
+OpenTelemetry.Instrumentation.GrpcNetClient.GrpcClientTraceInstrumentationOptions.CaptureRequestsHeaders.set -> void
+OpenTelemetry.Instrumentation.GrpcNetClient.GrpcClientTraceInstrumentationOptions.CaptureResponseHeaders.get -> System.Collections.Generic.IReadOnlyCollection<string!>?
+OpenTelemetry.Instrumentation.GrpcNetClient.GrpcClientTraceInstrumentationOptions.CaptureResponseHeaders.set -> void

Changes for OpenTelemetry.Instrumentation.Http package (if nullable enabled):

+OpenTelemetry.Instrumentation.Http.HttpClientTraceInstrumentationOptions.CaptureRequestsHeaders.get -> System.Collections.Generic.IReadOnlyCollection<string!>?
+OpenTelemetry.Instrumentation.Http.HttpClientTraceInstrumentationOptions.CaptureRequestsHeaders.set -> void
+OpenTelemetry.Instrumentation.Http.HttpClientTraceInstrumentationOptions.CaptureResponseHeaders.get -> System.Collections.Generic.IReadOnlyCollection<string!>?
+OpenTelemetry.Instrumentation.Http.HttpClientTraceInstrumentationOptions.CaptureResponseHeaders.set -> void

Changes for OpenTelemetry.Instrumentation.Owin package:

+OpenTelemetry.Instrumentation.Owin.OwinInstrumentationOptions.CaptureRequestsHeaders.get -> System.Collections.Generic.IReadOnlyCollection<string!>?
+OpenTelemetry.Instrumentation.Owin.OwinInstrumentationOptions.CaptureRequestsHeaders.set -> void
+OpenTelemetry.Instrumentation.Owin.OwinInstrumentationOptions.CaptureResponseHeaders.get -> System.Collections.Generic.IReadOnlyCollection<string!>?
+OpenTelemetry.Instrumentation.Owin.OwinInstrumentationOptions.CaptureResponseHeaders.set -> void

Option 2 - distinguish server and client options

Changes for OpenTelemetry.Instrumentation.AspNet package:

+OpenTelemetry.Instrumentation.AspNet.AspNetTraceInstrumentationOptions.CaptureRequestsServerHeaders.get -> System.Collections.Generic.IReadOnlyCollection<string!>?
+OpenTelemetry.Instrumentation.AspNet.AspNetTraceInstrumentationOptions.CaptureRequestsServerHeaders.set -> void
+OpenTelemetry.Instrumentation.AspNet.AspNetTraceInstrumentationOptions.CaptureResponseServerHeaders.get -> System.Collections.Generic.IReadOnlyCollection<string!>?
+OpenTelemetry.Instrumentation.AspNet.AspNetTraceInstrumentationOptions.CaptureResponseServerHeaders.set -> void

Changes for OpenTelemetry.Instrumentation.AspNetCore package (if nullable enabled):

+OpenTelemetry.Instrumentation.AspNetCore.AspNetCoreTraceInstrumentationOptions.CaptureRequestsServerHeaders.get -> System.Collections.Generic.IReadOnlyCollection<string!>?
+OpenTelemetry.Instrumentation.AspNetCore.AspNetCoreTraceInstrumentationOptions.CaptureRequestsServerHeaders.set -> void
+OpenTelemetry.Instrumentation.AspNetCore.AspNetCoreTraceInstrumentationOptions.CaptureResponseServerHeaders.get -> System.Collections.Generic.IReadOnlyCollection<string!>?
+OpenTelemetry.Instrumentation.AspNetCore.AspNetCoreTraceInstrumentationOptions.CaptureResponseServerHeaders.set -> void

Changes for OpenTelemetry.Instrumentation.GrpcNetClient package (if nullable enabled):

+OpenTelemetry.Instrumentation.GrpcNetClient.GrpcClientTraceInstrumentationOptions.CaptureRequestsClientHeaders.get -> System.Collections.Generic.IReadOnlyCollection<string!>?
+OpenTelemetry.Instrumentation.GrpcNetClient.GrpcClientTraceInstrumentationOptions.CaptureRequestsClientHeaders.set -> void
+OpenTelemetry.Instrumentation.GrpcNetClient.GrpcClientTraceInstrumentationOptions.CaptureResponseClientHeaders.get -> System.Collections.Generic.IReadOnlyCollection<string!>?
+OpenTelemetry.Instrumentation.GrpcNetClient.GrpcClientTraceInstrumentationOptions.CaptureResponseClientHeaders.set -> void

Changes for OpenTelemetry.Instrumentation.Http package (if nullable enabled):

+OpenTelemetry.Instrumentation.Http.HttpClientTraceInstrumentationOptions.CaptureRequestsClientHeaders.get -> System.Collections.Generic.IReadOnlyCollection<string!>?
+OpenTelemetry.Instrumentation.Http.HttpClientTraceInstrumentationOptions.CaptureRequestsClientHeaders.set -> void
+OpenTelemetry.Instrumentation.Http.HttpClientTraceInstrumentationOptions.CaptureResponseClientHeaders.get -> System.Collections.Generic.IReadOnlyCollection<string!>?
+OpenTelemetry.Instrumentation.Http.HttpClientTraceInstrumentationOptions.CaptureResponseClientHeaders.set -> void

Changes for OpenTelemetry.Instrumentation.Owin package:

+OpenTelemetry.Instrumentation.Owin.OwinInstrumentationOptions.CaptureRequestsServerHeaders.get -> System.Collections.Generic.IReadOnlyCollection<string!>?
+OpenTelemetry.Instrumentation.Owin.OwinInstrumentationOptions.CaptureRequestsServerHeaders.set -> void
+OpenTelemetry.Instrumentation.Owin.OwinInstrumentationOptions.CaptureResponseServerHeaders.get -> System.Collections.Generic.IReadOnlyCollection<string!>?
+OpenTelemetry.Instrumentation.Owin.OwinInstrumentationOptions.CaptureResponseServerHeaders.set -> void

Configuration by environmental variables

As the feature is needed for Automatic Instrumentation there is a need to configure options by environmental variables. There is no env. variables defined for this on the semantic convention level.

All packages except OpenTelemetry.Instrumentation.GrpcNetClient using environmental variables (directly by Environment.GetEnvironmentVariable) or by IConfiguration to configure its internal options.

There are 3 options to handled this

1. Add configuration option here.
2. Keep here only code-only options, implement env. variables in Automatic Instrumentation.

If decided to implement it here we should decide of the names

Follow Java naming convention

Use following set of variables

• OTEL_INSTRUMENTATION_HTTP_CLIENT_CAPTURE_REQUEST_HEADERS(Http and GrpcNetClient instrumentations)
• OTEL_INSTRUMENTATION_HTTP_CLIENT_CAPTURE_RESPONSE_HEADERS(Http and GrpcNetClient instrumentations
• OTEL_INSTRUMENTATION_HTTP_SERVER_CAPTURE_REQUEST_HEADERS (AspNet, AspNetCore, and Owin)
• OTEL_INSTRUMENTATION_HTTP_SERVER_CAPTURE_RESPONSE_HEADERS (AspNet, AspNetCore, and Owin)

Create .NET convention

Java convention does not follow the rules. It does on include OTEL_JAVA_ prefix. It should be done for all non-standard variables.

Variables for OpenTelemetry.Instrumentation.AspNet package:

• OTEL_DOTNET_TRACES_INSTRUMENTATION_ASPNET_CAPTURE_SERVER_REQUEST_HEADERS
• OTEL_DOTNET_TRACES_INSTRUMENTATION_ASPNET_CAPTURE_SERVER_RESPONSE_HEADERS

Variables for OpenTelemetry.Instrumentation.AspNetCore package:

• OTEL_DOTNET_TRACES_INSTRUMENTATION_ASPNETCORE_CAPTURE_SERVER_REQUEST_HEADERS
• OTEL_DOTNET_TRACES_INSTRUMENTATION_ASPNETCORE_CAPTURE_SERVER_RESPONSE_HEADERS

Variables for OpenTelemetry.Instrumentation.GrpcNetClient package:

• OTEL_DOTNET_INSTRUMENTATION_GRPCNETCLIENT_CAPTURE_CLIENT_REQUEST_HEADERS
• OTEL_DOTNET_INSTRUMENTATION_GRPCNETCLIENT_CAPTURE_CLIENT_RESPONSE_HEADERS

Variables for OpenTelemetry.Instrumentation.Http package:

• OTEL_DOTNET_TRACES_INSTRUMENTATION_HTTP_CAPTURE_CLIENT_REQUEST_HEADERS
• OTEL_DOTNET_TRACES_INSTRUMENTATION_HTTP_CAPTURE_CLIENT_RESPONSE_HEADERS

Variables for OpenTelemetry.Instrumentation.Owin package:

• OTEL_DOTNET_TRACES_INSTRUMENTATION_OWIN_CAPTURE_SERVER_REQUEST_HEADERS
• OTEL_DOTNET_TRACES_INSTRUMENTATION_OWIN_CAPTURE_SERVER_RESPONSE_HEADERS

Alternative approach

Implement all this stuff on the Automatic Instrumentation side. It is fully feasible as EnrichWith*Request and EnrichWith*Response are already available.

[Kielek]

@open-telemetry/dotnet-contrib-approvers, as it touches mostly core instrumentation package please let me know about your opinion.

I would like to implement both code-options and env. variable support in these packages.
If possible with dedicated env. naming per instrumentation package.

[vishweshbankwar]

@Kielek - For Instrumentation.AspNetCore,
Instrumentation.Http, we should wait for dotnet/aspnetcore#52439 and dotnet/runtime#93019 to avoid migration issues for users moving to .NET9.0 from lower versions.

OTEL_INSTRUMENTATION_HTTP_CLIENT_CAPTURE_RESPONSE_HEADERS and
OTEL_INSTRUMENTATION_HTTP_SERVER_CAPTURE_REQUEST_HEADERS - Are these part of spec?

[Kielek]

OTEL_INSTRUMENTATION_HTTP_CLIENT_CAPTURE_RESPONSE_HEADERS and
OTEL_INSTRUMENTATION_HTTP_SERVER_CAPTURE_REQUEST_HEADERS - Are these part of spec?

No, these variables are not part of spec nor semantic convention. It is just implemented on the Java side.

[rajkumar-rangaraj]

In addition to Vishwesh's input, here are my thoughts.

While I understand that this customization would make auto-instrumentation easier, I believe we should avoid introducing such customizations directly into the instrumentation library. Creating new APIs or environment variables for this purpose may not be the best option due to the potential complexity and maintenance overhead it would introduce.

Instead, I recommend exploring alternative approaches that leverage existing mechanisms within the OpenTelemetry framework. One such approach could be to use the enrich API in the instrumentation options to handle the specific headers you need. This way, the core library remains lean and focused, while still allowing for the flexibility required by specific use cases.

For auto-instrumentation, we could explore a design to incorporate it as a part of it by customizing instrumentation options.

[Kielek]

@rajkumar-rangaraj, I can agree that the env. vars. are only importand for auotmatic instrumentation and there is not need to put it directly in the instrumentation library itself, but using Enrich* functions is not developer friendly for any standard, Opt-In attributes (all this attributes are part of the semantic convention).

[rajkumar-rangaraj]

If the Enrich* functions are not developer-friendly and are causing difficulties, our priority should be to address and improve their usability within the instrumentation libraries rather than seeking alternative solutions.

[CodeBlanch]

Piggybacking on what @vishweshbankwar said above...

IMO let's stick with using Enrich for now. We kind of need to wait and see what happens with .NET9 and the planned tracing updates. I would hate to add a bunch of features now that we then tell users in ~Nov won't work on .NET9 or they have to go and change their code to get the same results. Once we see those new APIs we'll have to figure out what to do with our instrumentation libraries. Ideally we'll be able to bridge the existing Enrich APIs onto whatever runtime provides so that users upgrading to .NET9 won't have to change anything. Could be difficult, who knows 🤣

[Kielek]

Sure, we will go then with implementation in auto-instrumentation side. Hopefully, we will be able to contribute it here in November. If you have any beta-version of implementation I will be happy to verify if it covers all supported cases by AutoInstumentation.

[Kielek]

Functionality implemented on Automatic Instrumentation side.
Closing issue here. Please reopen, when you think that we can bring functionalities here.

[HeenaBansal20]

@Kielek , does this opt in variables work for php instrumentation too
?

[Kielek]

@HeenaBansal20, I think that the best way is to ask in https://github.com/open-telemetry/opentelemetry-php-contrib (creating issue).

I do not know basically anything about PHP instrumentation.