[Monitor Opentelemetry] Duplicated dependencies span when enabling azure-sdk + http instrumentation

[Apokalypt]

• Package Name: @azure/monitor-opentelemetry
• Package Version: 1.8.0
• Operating system: MacOS + Windows
• nodejs
  • version: 18.20.4 + 20.17.0 + 22.9.0
• browser
  • name/version:
• typescript
  • version:
• Is the bug related to documentation in
  • README.md
  • source code documentation
  • SDK API docs on https://learn.microsoft.com

Describe the bug
Since the migration to OpenTelemetry with @azure/opentelemetry-instrumentation-azure-sdk + @opentelemetry/instrumentation-http we are seing duplicated span when an HTTP call is emitted from an azure SDK libraries :

• First one is created from the tracingPolicy of the package @azure/core-rest-pipeline
• Second one is created from the @opentelemetry/instrumentation-http which offer better customisation for outgoing request (ignore or not, custom attributes from headers, .......)

To Reproduce
Steps to reproduce the behavior:

1. Run the below code

const { useAzureMonitor } = require('@azure/monitor-opentelemetry')

useAzureMonitor({
    instrumentationOptions: {
        http: { enabled: true },
        azureSdk: { enabled: true },

        mongoDb: { enabled: false },
        redis4: { enabled: false },
        mySql: { enabled: false },
        redis: { enabled: false },
        bunyan: { enabled: false },
        winston: { enabled: false },
        postgreSql: { enabled: false }
    },
    browserSdkLoaderOptions: { enabled: false }
})

const { DefaultAzureCredential } = require('@azure/identity')
const { SecretClient } = require('@azure/keyvault-secrets')
const { trace } = require('@opentelemetry/api')

const client = new SecretClient(`https://${process.env.KEYVAULT_NAME}.vault.azure.net`, new DefaultAzureCredential())

trace.getTracer('name', 'version').startActiveSpan('test_duplicated_span', async (span) => {
    console.warn('Getting secret key...', span.spanContext().traceId)

    const key = 'StorageConfName'
    const result = await client.getSecret(key)
    console.log(result)
})

setTimeout(() => {
    console.log('Waiting for everything to be sent to Application Insights...')
}, 100_000)

Expected behavior
Only one span should be created for an HTTP call even if he the request goes through the tracing policy. For that there is different possibilities :

• Azure tracing policy should not generate a span for HTTP calls when @opentelemetry/instrumentation-http is enabled
• Azure Monitor + @azure/opentelemetry-instrumentation-azure-sdk should expose options to indicate if HTTP calls should be tracked or not
• Azure tracing policy should be compliant with @opentelemetry/instrumentation-http in terms of property/status/... If that's the case we can easily ignore request emitted from Azure SDK libraries while being able to keep all features (custom span attributes, ignore some requests, ....)

Screenshots

Additional context
Add any other context about the problem here.

[JacksonWeber]

@Apokalypt while exclusive behavior is not automatic when instrumenting with the http instrumentation yet, you can use the AZURE_TRACING_DISABLED environment variable to disable all Azure traces.

[Apokalypt]

@Apokalypt while exclusive behavior is not automatic when instrumenting with the http instrumentation yet, you can use the AZURE_TRACING_DISABLED environment variable to disable all Azure traces.

For the moment we've decided to keep the traces even if we have duplicated HTTP calls, because the other spans are quite useful. But, anyway, thanks for the tip

Can we expect an implementation of something that avoids this in the near future or not?

[JacksonWeber]

@Apokalypt if you're looking to keep the parent Azure tracing spans, but not duplicate the spans from the http-instrumentation, you should be able to use the AZURE_HTTP_TRACING_CHILDREN_DISABLED environment variable then.

As for automatic detection, I'll investigate.

[Apokalypt]

@Apokalypt if you're looking to keep the parent Azure tracing spans, but not duplicate the spans from the http-instrumentation, you should be able to use the AZURE_HTTP_TRACING_CHILDREN_DISABLED environment variable then.

As for automatic detection, I'll investigate.

I didn't know about this parameter, but it doesn't suit us anyway, so we'll stick with duplicate traces for now. In fact, our configuration of the http-instrumentation library is not the default because we needed to add certain headers to the span and ignore others. Therefore, we don't want to remove traces/spans generated by this library

As for automatic detection, if this is not possible, we should at least be able to configure/customize the behavior, just as the http-instrumentation library does ( documentation ) This would certainly be easier to implement, and would also make it easier to let developers choose what span they want generated by Azure libraries.

[JacksonWeber]

@Apokalypt I'm a bit confused then. From your issue your goal appears to be disabling the duplicate http instrumentation traces coming from the Azure SDK's automatic tracking. Brining your own configuration of the OpenTelemetry http-instrumentation should be fine assuming you use that instrumentation configured to your liking + the Azure Monitor OpenTelemetry distro with the above environment variable enabled.

[Apokalypt]

@JacksonWeber Well, I'll try to rephrase to make sure everything's clear.

The problem is that if you enable both HTTP and Azure instrumentation, you get duplicate traces for each HTTP call. So we'd like a solution that avoids this duplication (while preserving the other traces generated by the Azure instrumentation library, e.g. SecretClient.getSecret from my screen), while preserving the additional data we add to the HTTP traces generated by the “http-instrumentation” package (e.g. rate limit headers from response).
That's why the above environment variables are not enough :

• AZURE_HTTP_TRACING_CHILDREN_DISABLED -> it removes traces generated by http-instrumentation and therefore we are loosing additional data from our custom configuration of the package
• AZURE_TRACING_DISABLED -> it removes all traces generated by azure instrumentation package while we would like to keep traces that are not HTTP (e.g. SecretClient.getSecret)

---

On the other hand, what I meant in my previous message was that we'll adapt to whatever solution is proposed by azure instrumentation package. I'm thinking of the following solutions:

• Automatic detection of the “http-instrumentation” package so as not to generate HTTP traces in “azure-instrumentation”.
• Adding a possible configuration to say that “azure-instrumentation” must not generate traces for HTTP calls, e.g.

const { useAzureMonitor } = require('@azure/monitor-opentelemetry')

useAzureMonitor({
    instrumentationOptions: {
        http: { enabled: true, ...customHttpInstrumentationOptions },
        azureSdk: { enabled: true, generateHttpTraces: false }
    }
})

• Align the possible configuration of “azure-instrumentation” with the one of “http-instrumentation” to enable us to add the additional data to the HTTP span generated by “azure-instrumentation”. If this solution is selected, we can avoid duplicate traces by customizing “http-instrumentation” package