feat: add tracing to query path

[waynr]

This propagates a TraceCollector to the query call path from the common server state. With this we should get many of the same traces that we see in IOx since there is a lot of shared code on that call path.

I have not yet been able to see traces show up in the jaeger instance that this connects to so it seems like there must still be something missing in the setup. My setup is more or less to run jaeger in docker:

  docker run -d \
    --name jaeger \
    -p 6831:6831/udp \
    -p 16686:16686 \
    jaegertracing/all-in-one:latest

Then to run influxdb3 serve ... as usual with the following environment variables set:

export TRACES_EXPORTER=jaeger
export TRACES_EXPORTER_JAEGER_AGENT_HOST=localhost
export TRACES_EXPORTER_JAEGER_AGENT_PORT=6831

Once I've done this, I use the influxdb3_load_generator to generate write and query loads, observe logs in the server showing that the writes and queries are getting through, but don't see any traces in the jaeger web console at localhost:16686.

I'm opening this PR as a draft to get eyes from others on the team while I
continue digging around.

[hiltontj]

I have not yet been able to see traces show up in the jaeger instance that this connects to so it seems like there must still be something missing in the setup. [...]

@MaduMitha-Ravi - do you notice anything different between the environment/jaeger setup described above and how the perf team is collecting traces?

@waynr the only obvious thing I can point out is that the QueryExecutor::query_influxql method will probably also need a similar change - though, I am fairly certain the load generator is using SQL, so that isn't likely the reason that you're not seeing anything collected.

[MaduMitha-Ravi]

@saraghds any thoughts?

[saraghds]

We run our own jaeger instance on our machines and set the env vars during docker run:

docker run -d --name jaeger-1.65 \
	 \
	-e TRACES_EXPORTER=jaeger -e TRACES_EXPORTER_JAEGER_AGENT_HOST=10.92.104.1 \
	-e TRACES_EXPORTER_JAEGER_AGENT_PORT=6831 -e TRACES_EXPORTER_JAEGER_TRACE_CONTEXT_HEADER_NAME=influx-trace-id\
	--log-driver=grafana/loki-docker-driver:2.9.2 \
	--log-opt "loki-url=http://10.92.104.1:3100/loki/api/v1/push" \
	--log-opt "loki-external-labels=job=jaeger-1.65,container={{.ID}}" \
	-p 6833:6831/udp \
-p 4319:4317 \
-p 16687:16686 \
-v /opt/perf-tools:/opt/perf-tools jaegertracing/all-in-one:1.65.0 --sampling.strategies-file=/opt/perf-tools/etc/jaeger-strategies.json

[waynr]

Okay this took me an embarrassingly long time to figure out, but the problem appears to have been that Jaeger listens for Thrift protocol on a different port in v 1.x than in v 2.x.

When I tried using either of the suggested containers:

• jaegertracing/all-in-one:latest from https://github.com/influxdata/influxdb_iox/blob/main/docs/tracing.md#step-1-run-jaeger-locally
• jaegertracing/all-in-one:1.65.0 from @saraghds's comment

I was using Jaeger v1, which listens for thrift over http at port 14268, but had the port mapping -p 6831:6831/udp set on my docker command, there was no way for the UDP packets to reach port 14268. The Jaeger executor in influxdb3 didn't complain, possibly because it looked like a connection was made since the port mapping was up. tcpdump also showed packets being sent to the container process, but Jaeger never showed any sign that it was receiving the packets when I had the log level set to debug.

Once I switched over to jaegertracing/jaeger:2.3.0, I started seeing traces:

So it looks like this PR as-is gets us to the point of receiving some of the same spans as IOx, but there's more work to be done to get all of the spans requested in #25911

[hiltontj]

FWIW, the change in this PR looks like a win to me and should be helpful to the perf team.