Which nginx instrumentation is the future?

[mihn]

Hello,
Can you tell me which nginx instrumentation nginx or otel-webserver-module is going to be supported in the future - because I cannot figure it out on my own.
nginx one has features that I like (ability to block incoming span), but otel-webserver-module is featured on OTel Blog and suportes latest nginx releases.
So which is it?

Thank you for your attention.

[esigo]

otel-webserver-module is the one that is actively maintained now.

[mihn]

Thank you for clarification.
Any plans to bring features from nginx to otel-webserver-module ? Especially variables, support for none collector and feature similiar to what opentelemetry_trust_incoming_spans does?

[DebajitDas]

Thanks @mihn for raising this concern. We would definitely look at the viability of the features in otel-webserver-module.
Request you to list down all the features that you want to see here?

[phillipuniverse]

Other feature would be all the other supported nginx directives like capturing incoming headers as attributes or trusting incoming trace/span ids.

[mihn]

Anytime @DebajitDas
Here's the list:

1. Decoupling enabling the telemetry from exporting it (supporting "none" exporter) - nginx should be able to enrich the requests, but without exporting them to external system.
2. Define nginx variables for traceId and spanId that would be usable in, for example, logfiles.
3. Ability to distrust incoming span.

[DebajitDas]

@mihn I am clear on point 1 and point 3. I request you to please provide some more details related to point 2 and its use-cases. Does nginx provide this feature?

[phillipuniverse]

Does nginx provide this feature?

@DebajitDas yes, see https://github.com/open-telemetry/opentelemetry-cpp-contrib/blob/main/instrumentation/nginx/README.md#nginx-variables.

The main use case I use this for is to include the trace id and span id in the nginx access log.

[mihn]

Same case for me - log traceId/spanId in every access log entry :)

[mhosabettu]

Hi @mihn ,

Could you help us by explaining a scenario where 1 and 3 would be useful for a customer? How would supporting "none" exporter help a customer? or in what situations does one want to distrust incoming span?

[mihn]

1. Let's say that it would be generally helpful, because there might be sampling applied when exporting to remote backend, and errors might not be exported due to sampling.
2. I'm thinking using it on boundaries - 3rd party calling A or C calling 3rd party. From personal experience I know organizations that would love to have both.

[phillipuniverse]

Could you help me think of a scenario where one doesn't want to monitor A and wants to start monitoring from B->C?

Think of the case where you run https://api.myorg.com. Requests are handled at first by nginx acting as an api gateway which then proxies to request to some backend service. I want nginx to start the trace/span so that nginx -> backend service is monitored.

However, if some random client sends a valid traceparent header to https://api.myorg.com, I want to ignore that since it would be coming from outside my organization. Admittedly, this is a bit of an assumption on my part that trusting a random incoming trace/span id is not a good idea, maybe at the end of the day it doesn’t really matter?

[phillipuniverse]

Since this feature will be helpful in dev/test/staging env, would you manually log in to the system to read logs in case of errors?

Yes, or wherever you are capturing logs. In my org we run in ECS so all logs are already exported to Cloudwatch. I would log into Cloudwatch and filter by a trace id.

We export our traces/spans to Datadog and just like @mihn mentioned, Datadog APM is pricy so we turn off exports in our staging environments. But there is utility in having a common trace id in logs and other error reporting systems to give us some level of traceability.

[WojciechKuk]

in what situations does one want to distrust incoming span?

I have such needs. I want to generate traceIds with some control codes, and send them to frontend (opentelemetry-js). Then I need to accept only "my" traces returning from the frontend. Not random traces generated by i.e. some botnet, or other "bad actor".

[mhosabettu]

Thank you @WojciechKuk

[phillipuniverse]

If the otel-webserver-module is the future, a possible way forward to avoid further confusion would be to:

1. Go through the current nginx directives and current Apache directives
2. Produce a delta with what is or isn’t supported in the otel-webserver-module at the moment
3. Provide a roadmap for inclusion of missing features or a decision on why support for a directive will not be added
4. Officially deprecate the nginx and Apache instrumentation once otel-webserver-module has reached parity
5. Some time later, remove the existing instrumentation code

[mhosabettu]

Thank you for providing detailed explanations @mihn and @phillipuniverse. Let me discuss with the team and get back on features we would be adding to otel-webserver-module in the upcoming months.

[mhosabettu]

We have decided to add most of the missing directive support to otel webserver agent. Additionally we will also be looking at adding variables and missing attributes support

[mihn]

Thank you @mhosabettu.

[4141done]

I'd like to add my support here for inclusion of the opentelemetry nginx directives in the webserver module. I'd especially like to see opentelemetry_operation_name since currently I get a lot of spans related to the various core module operations but nothing that I can semantically link to a certain operation in my overall trace.

[phillipuniverse]

@DebajitDas it's been ~4mo since this discussion was started, just checking in to see if you all have identified the gaps and what remains?

I would like to contribute but the first step would be to compile the differences between the 2 so we can start to check them off.

[DebajitDas]

Hi @phillipuniverse,

We had identified the gaps that are to be added here. But due to other priorities and bandwidth crunch we were not able to work on this as of now.
Following are the gaps that are identified:

1. Trust incoming spans
2. opentelemetry_attribute
3. opentelemetry_context_traceparent
4. b3+ opentelemetry_propagate
5. opentelemetry_config
6. opentelemetry_operation_name
7. opentelemetry_ignore_paths

We had a plan to complete this by April End, but due to other business needs these went to our backlog.
Please feel free to contribute and we would review.

[ajaynagariya]

Hi phillipuniverse
Team is going to review the enhancement and other activities as part of OTEL Webserver Nginx sometimes in May month and would update team here with timelines for next release/updates

[dmiyamasu]

I'd like to propose a specific clarification of the otel-webserver-module including support for NGINX running on alpine linux. There is an open issue requesting more support for alpine, and if we can roll NGINX into that feature request, we could kill two birds with one stone.

[holmofy]

Another nginx-otel maintained by nginx