Errors in traces merely showing error=true

[abernix]

In some cases, when errors are encountered all that appears in Datadog's Errors is an error=true property on the span.

[Geal]

it looks like the issue here is simply that opentelemetry-datadog does not know how to send logs to datadog: open-telemetry/opentelemetry-rust-contrib#6

It looks like sending the spans to the ddog agent using OTLP works though: the error events appear in the span's metadata

[Meemaw]

I can confirm this behaviour when using otlp exporter sending to a datadog agent opentelemtry collector. All error spans just have the error=true and no context about the error whatsoever.

[Geal]

the opentelemetry events will appear under the events field of the span metadata:

That is not a great experience though, but for now we cannot do much about it, because the opentelemetry-datadog crate does not support Datadog's logs API (which is separated from the traces), and even Datadog's own agent pretends to understand OTLP but will not send the events as logs.

[mohitd11]

Is there any workaround for this?

[Geal]

Not yet unfortunately, I have not figured out how to correlate logs properly. A thing that could be tested: get logs from stdout, process them through vector to translate the otel trace ids to datadog trace ids, then send to datadog

[chandrikas]

@BrynCooke, can you please check this is still an issue?

[BrynCooke]

This is still an issue, however we cannot do anything about this and it needs to be fixed upstream.
It looks like Datadog are trying to migrate their collector to use code from the official otel Go codebase which hopefully will make exporting to Datadog via OTLP an option.

[abernix]

It's possible that #3537 might fix this? Thoughts?

[psbrandt]

I am not seeing my events in the span "info" section in Datadog. I used to see them, but don't anymore. Did something change? Or maybe it's a Datadog issue? Is anyone else having the same experience?

EDIT: Coming back to answer my own question. This is the (relatively new) code that is preventing events from showing up in Datadog:

router/apollo-router/src/plugins/telemetry/reload.rs

Lines 178 to 181 in 6bb450b

	// we ignore events
	if !meta.is_span() {
	return false;
	}

[BrynCooke]

It looks like #2999 has compounded this issue, filtering out all events from spans. Opened #4321.

[Geal]

this should be revisited now that #4486 is merged, it should be fixed