Not able to link Trace to Profile with "otel-profiling-java" package

[Vaibhav-1995]

Bug

As per below document once we add "otel-0.11.0.jar" in application image it will automatically assign profiles with span IDs and with that after making mentioned setting in Tempo data source we will be able to see Trace to Profile linkage.
https://grafana.com/docs/pyroscope/latest/configure-client/trace-span-profiles/java-span-profiles/

But it is not working in our setup since our application image is build through CI-CD pipeline and when we add below dependency and build our image the "otel-0.11.0.jar" gets added in "app.jar" in application image and not as separate jar file in any of image directory. Due to which this setup is not working in our case and not able to link Trace to Profile.

<dependency>
    <groupId>io.pyroscope</groupId>
    <artifactId>otel</artifactId>
    <version>0.11.0</version>
</dependency>

Also added ENV OTEL_JAVAAGENT_EXTENSIONS=./pyroscope-otel.jar in application deployment but still that jar is not getting started from app.jar.

Expected Behavior

As mentioned in document, we should be able to link Trace to Profile once "otel-0.11.0.jar" is added to app image irrespective of how it gets added (in this case it is getting added in app.jar - when extracted app.jar it is present on path /BOOT-INF/lib/otel-0.11.0.jar)

Environment

• Infrastructure: Kubernetes
• Deployment tool: helm

Additional Context

As tested same is working locally and able to link Trace to Profile if we create Dockerfile and add "otel-0.11.0.jar" mannually as mention in documentation beacuse it gets added as a separate jar file on mentioned path and also ENV OTEL_JAVAAGENT_EXTENSIONS=./pyroscope-otel.jar starts it.

Question

How can we achieve this with mentioned setup as our app image is getting build through pipeline and otel-0.11.0.jar in geeting added in app.jar ? Also how can we start that jar which is in app.jar on path /BOOT-INF/lib/otel-0.11.0.jar ?

[Vaibhav-1995]

Hi Team, Any update?

[Vaibhav-1995]

Hi Team,

Could you please check on this? We are stuck on it.

[aleks-p]

Hi @Vaibhav-1995

I don't think this is directly possible. What could work is building a custom OpenTelemetry auto-instrumentation jar that has the extension bundled inside. Then, you should be able to avoid adding it to your application entirely. We haven't tried this though, and we don't provide this jar currently.

You can find some info and guidance on this here: https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/5ffaddc0ed6fec594e88de79a976dc8f10ae799f/examples/extension/README.md?plain=1#L29, if you want to try to build such an agent yourself.

Here is where the loading of embedded extensions happens:
https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/5ffaddc0ed6fec594e88de79a976dc8f10ae799f/javaagent-tooling/src/main/java/io/opentelemetry/javaagent/tooling/ExtensionClassLoader.java#L79

In the last link you'll notice that the extension jar is extracted from the agent jar to a temporary location and loaded as usual. If you have some flexibility in your setup, you could facilitate a similar approach as well.

[Vaibhav-1995]

Thanks @aleks-p

Will check one above approch if it's possible from our end and will get back on it.

Additionaly, to make it simplier do you have any idea if we can bundle this jar directly with java-autoinstrumentation image in docker file and can use it in any way as we are already using java-autoinstrumentation for tracing so that image is getting injected in app pod and also it injects javaagent.jar by creating otel-auto-instrumentation-java directory.

So if we can add otel-0.11.0.jar in that image and can inject it in same otel-auto-instrumentation-java directory, I think it may work.