[Feature]: `opentelemetry-appender-tracing` should respect information inside `tracing::Span`

[0xForerunner]

It's great to be able to correlate logs with traces as is possible with the Grafana Tempo+Loki combo. This could be some sort of configuration option likely.

[cijothomas]

The logs should be correlated automatically.. Are you not seeing that behavior?

[lalitb]

Yes - this should be correlated automatically, as long as the application is instrumented using opentelemetry tracing (either through tracing-opentelemetry or directly using opentelemetry tracing api). Not if it is using any other (non-otel) tracing subscriber.

[0xForerunner]

This is the information I'm getting in loki, with no trace_id information.

Logs are being generated with this code

        let logs = opentelemetry_otlp::new_pipeline()
            .logging()
            .with_exporter(opentelemetry_otlp::new_exporter().http())
            .with_log_config(
                opentelemetry_sdk::logs::config().with_resource(Resource::new(vec![
                    KeyValue::new("service.name", "liquidation-bot-rs"),
                ])),
            )
            .install_batch(opentelemetry_sdk::runtime::Tokio)?;

        let logging = OpenTelemetryTracingBridge::new(&logs.provider().unwrap())
            .with_filter(EnvFilter::from_default_env());
        
        subscriber = Registry::default()
            .with(stdout)
            .with(logging)
            .with(paper_trail)
            .with(metrics)
            .with(tracing)

[0xForerunner]

Yes - this should be correlated automatically, as long as the application is instrumented using opentelemetry tracing (either through tracing-opentelemetry or directly using opentelemetry tracing api). Not if it is using any other (non-otel) tracing subscriber.

I can confirm things are properly instrumented with tracing-opentelemetry. The log in the screenshot above is inside a function with #[instrument] and I am correctly seeing traces and events.

[cijothomas]

Can you use stdout exporter and share the output? We just want to confirm if this issue is with OTLP or Loki itself.

[cijothomas]

Can you use stdout exporter and share the output? We just want to confirm if this issue is with OTLP or Loki itself.

Or just the log output from otel collector

[0xForerunner]

Can you use stdout exporter and share the output? We just want to confirm if this issue is with OTLP or Loki itself.

This code somehow results in no output...

        let exporter = opentelemetry_stdout::LogExporter::default();
        let logs = LoggerProvider::builder()
            .with_simple_exporter(exporter)
            .build();
        let logging = OpenTelemetryTracingBridge::new(&logs);

[0xForerunner]

collector logs are scarce

otel-collector] [2023-11-16 07:19:57] 2023-11-16T07:19:57.702Z info    service@v0.88.0/telemetry.go:84 Setting up own telemetry...
[otel-collector] [2023-11-16 07:19:57] 2023-11-16T07:19:57.702Z info    service@v0.88.0/telemetry.go:201        Serving Prometheus metrics      {"address": ":8888", "level": "Basic"}
[otel-collector] [2023-11-16 07:19:57] 2023-11-16T07:19:57.703Z info    lokiexporter@v0.88.0/exporter.go:41     using the new Loki exporter     {"kind": "exporter", "data_type": "logs", "name": "loki"}
[otel-collector] [2023-11-16 07:19:57] 2023-11-16T07:19:57.707Z info    service@v0.88.0/service.go:143  Starting otelcol-contrib...     {"Version": "0.88.0", "NumCPU": 8}
[otel-collector] [2023-11-16 07:19:57] 2023-11-16T07:19:57.708Z info    extensions/extensions.go:33     Starting extensions...
[otel-collector] [2023-11-16 07:19:57] 2023-11-16T07:19:57.708Z info    extensions/extensions.go:36     Extension is starting...        {"kind": "extension", "name": "basicauth/server"}
[otel-collector] [2023-11-16 07:19:57] 2023-11-16T07:19:57.708Z info    extensions/extensions.go:43     Extension started.      {"kind": "extension", "name": "basicauth/server"}
[otel-collector] [2023-11-16 07:19:57] 2023-11-16T07:19:57.708Z info    extensions/extensions.go:36     Extension is starting...        {"kind": "extension", "name": "basicauth/traces"}
[otel-collector] [2023-11-16 07:19:57] 2023-11-16T07:19:57.708Z info    extensions/extensions.go:43     Extension started.      {"kind": "extension", "name": "basicauth/traces"}
[otel-collector] [2023-11-16 07:19:57] 2023-11-16T07:19:57.708Z info    extensions/extensions.go:36     Extension is starting...        {"kind": "extension", "name": "basicauth/metrics"}
[otel-collector] [2023-11-16 07:19:57] 2023-11-16T07:19:57.708Z info    extensions/extensions.go:43     Extension started.      {"kind": "extension", "name": "basicauth/metrics"}
[otel-collector] [2023-11-16 07:19:57] 2023-11-16T07:19:57.708Z info    extensions/extensions.go:36     Extension is starting...        {"kind": "extension", "name": "basicauth/logs"}
[otel-collector] [2023-11-16 07:19:57] 2023-11-16T07:19:57.708Z info    extensions/extensions.go:43     Extension started.      {"kind": "extension", "name": "basicauth/logs"}
[otel-collector] [2023-11-16 07:19:57] 2023-11-16T07:19:57.709Z warn    internal@v0.88.0/warning.go:40  Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks        {"kind": "receiver", "name": "otlp", "data_type": "metrics", "documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"}
[otel-collector] [2023-11-16 07:19:57] 2023-11-16T07:19:57.709Z info    otlpreceiver@v0.88.0/otlp.go:101        Starting HTTP server    {"kind": "receiver", "name": "otlp", "data_type": "metrics", "endpoint": "0.0.0.0:4318"}
[otel-collector] [2023-11-16 07:19:57] 2023-11-16T07:19:57.710Z info    service@v0.88.0/

[cijothomas]

Follow this example as-is https://github.com/open-telemetry/opentelemetry-rust/blob/main/opentelemetry-otlp/examples/basic-otlp/README.md#manual

You should see logs correctly correlated to traces
https://github.com/open-telemetry/opentelemetry-rust/blob/main/opentelemetry-otlp/examples/basic-otlp/src/main.rs#L126

[0xForerunner]

Can you use stdout exporter and share the output? We just want to confirm if this issue is with OTLP or Loki itself.

This code somehow results in no output...

        let exporter = opentelemetry_stdout::LogExporter::default();
        let logs = LoggerProvider::builder()
            .with_simple_exporter(exporter)
            .build();
        let logging = OpenTelemetryTracingBridge::new(&logs);

Do you know what's going on here? Something about that is blocking all output on the tracing subscriber.

[cijothomas]

collector logs are scarce

I should have been more precise, apologies. You need "logging" exporter in collector to see the actual data it receives.
https://github.com/open-telemetry/opentelemetry-rust/blob/main/opentelemetry-otlp/examples/basic-otlp/otel-collector-config.yaml#L14

[cijothomas]

Can you use stdout exporter and share the output? We just want to confirm if this issue is with OTLP or Loki itself.

This code somehow results in no output...

        let exporter = opentelemetry_stdout::LogExporter::default();
        let logs = LoggerProvider::builder()
            .with_simple_exporter(exporter)
            .build();
        let logging = OpenTelemetryTracingBridge::new(&logs);

Do you know what's going on here? Something about that is blocking all output on the tracing subscriber.

Start with the example from the repo. That should work as-is. And you can tweak it to match your setup.
https://github.com/open-telemetry/opentelemetry-rust/blob/main/opentelemetry-appender-tracing/examples/basic.rs