How should we configure our "default" opentelemetry subscriber?

[garypen]

Right now for each graphql request, we try to get a tracing subscriber from the supplied configuration. If no configuration is provided, i.e.: not using otlp or jaeger, then we end up with the "default" subscriber.

    let dispatcher = configuration
        .subscriber
        .clone()
        .map(tracing::Dispatch::new)
        .unwrap_or_default();

I think this is then created from the default subscriber we set in main.rs where we initialise a JSON subscriber. The question here is "do we want the subscriber configured for main binary logging to be the same subscriber we use for library tracing?". I think not, but it's something we need to decide.

I think a better approach would be to configure a "sane" subscriber to use if there is no configured oltp or jaeger configuration provided. Almost certainly not using a json formatter. Something based on the GLOBAL_ENV_FILTER.
Something like:

    let dispatcher = match &configuration.subscriber {
        Some(s) => tracing::Dispatch::new(s.clone()),
        None => tracing::Dispatch::new(
            tracing_subscriber::fmt::fmt()
                .with_env_filter(
                    GLOBAL_ENV_FILTER
                        .get()
                        .expect("GLOBAL_ENV_FILTER must be set to a valid value"),
                )
                .finish(),
        ),
    };

(Ideally, we wouldn't do this processing each time we call run_graphql_request(), but would hoist it and just do it when we process configuration and hold a non-optional dispatcher in configuration.)

What do people think? (and is my understanding of the way logging/tracing works correct?)

[Geal]

Currently we can have both the JSON subscriber and the Jaeger/OTLP one working at the same time. The JSON subscriber gets the events like tracing::info!() while OTLP or Jaeger get the spans and the events.

I think the subscriber should be created only once (not on every request like we do right now) and keep the fallback on the env filter (right now we setup the JSON logger with env filter, then try to add the OTLP one, and return None if the OTLP subscriber creation failed, while we should just return the JSON one)

[Geal]

FYI I'm solving this in #224