[Bug]: Ballerina traces not listed as a child in API Manager

[NipunaMadhushan]

Description

There is an example http service that has implemented observable and jaeger modules using otel collector, and is exposed by traefik and wso2 apim 4.4.0. Ballerina service has been exposed as an endpoint in an API exposed through APIM.
I can see the trace traefik -> apim as child, but ballerina is in a different trace.
Ballerina service should appear as a child of traefik and apim.

ballerina: started publishing traces to Jaeger on localhost:4317
time=2025-01-07T06:10:07.776-06:00 level=INFO module=wso2/albums message="Accept:/" traceId="66be1a05f0d3dccb5c7b1940606d94b2" spanId="47d5672b7245617b"
time=2025-01-07T06:10:07.828-06:00 level=INFO module=wso2/albums message="Accept-Encoding:gzip, deflate, br" traceId="66be1a05f0d3dccb5c7b1940606d94b2" spanId="47d5672b7245617b"
time=2025-01-07T06:10:07.838-06:00 level=INFO module=wso2/albums message="activityid:edd9ef87-83cd-409e-a4b7-74bbf97a738b" traceId="66be1a05f0d3dccb5c7b1940606d94b2" spanId="47d5672b7245617b"
time=2025-01-07T06:10:07.857-06:00 level=INFO module=wso2/albums message="Connection:Keep-Alive" traceId="66be1a05f0d3dccb5c7b1940606d94b2" spanId="47d5672b7245617b"
time=2025-01-07T06:10:07.887-06:00 level=INFO module=wso2/albums message="Host:localhost:9090" traceId="66be1a05f0d3dccb5c7b1940606d94b2" spanId="47d5672b7245617b"
time=2025-01-07T06:10:07.899-06:00 level=INFO module=wso2/albums message="Traceparent:00-28a927a37a6531305958e5253eeffc64-1974eab72c4e43ea-01" traceId="66be1a05f0d3dccb5c7b1940606d94b2" spanId="47d5672b7245617b"
time=2025-01-07T06:10:07.911-06:00 level=INFO module=wso2/albums message="User-Agent:Synapse-PT-HttpComponents-NIO" traceId="66be1a05f0d3dccb5c7b1940606d94b2" spanId="47d5672b7245617b"

The configurations are as follows.
Config.toml -

[ballerina.observe]
tracingEnabled=true
tracingProvider="jaeger"

[ballerinax.jaeger]
agentHostname="localhost"
agentPort=4317
samplerType="const"
samplerParam=1.0
reporterFlushInterval=2000
reporterBufferSize=1000

deployment.toml -

[apim.open_telemetry]
remote_tracer.enable = true
remote_tracer.name = "otlp"
remote_tracer.url = "http://otel-otel-collector-1:4317/v1/traces"

[[apim.open_telemetry.remote_tracer.properties]]
name = "apim"
value = "4.4.0"

Steps to Reproduce

No response

Affected Version(s)

Ballerina version - 2201.10.3

OS, DB, other environment details and versions

APIM version - 4.4.0

Related area

-> Observability

Related issue(s) (optional)

No response

Suggested label(s) (optional)

No response

Suggested assignee(s) (optional)

No response

[NipunaMadhushan]

Seems like the headers provided from API Manager are not been processed by Ballerina service. Therefore, Spans related to Ballerina service has been nested under a separate trace.

[NipunaMadhushan]

There are multiple ways to publish span data into Jaeger server via APIs.
https://www.jaegertracing.io/docs/2.3/apis/#write-apis

APIM uses Legacy Protobuf format via 14250 port to publish span data which is deprecated now (https://www.jaegertracing.io/docs/2.3/apis/#legacy-protobuf-via-grpc).

Ballerina use OTLP Protobuf format via 4317 port to publish span data. These format differences may cause to identify headers related to parent-child relation between spans of APIM and Ballerina service.

[NipunaMadhushan]

Tested with zipkin and it works fine as Ballerina service span nested under APIM span as a child span.

[NipunaMadhushan]

Found the issue. Issue lies with the propagation where APIM uses JaegerPropagator. Ballerina previously use JaegerPropagator for jaeger extension (upto 0.5.2 version) and changed to W3CContextPropagator recently. This caused this issue.

With Jaeger version 0.5.2, This works fine. As a workaround, you can use the jaeger version as 0.5.2 and lock the dependencies by running bal run --sticky.