SERVER_ADDR
- server bind address inhost:port
format; defaults to0.0.0.0:8080
.BASE_URL_PATH
- URL path prefix to be applied to all public API endpoints; defaults to/api/v0
OpenTelemetry library configuration is done with the standardized environment variables listed here here and [here] (https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/).
Of particular interest are:
OTEL_EXPORTER_OTLP_ENDPOINT
- this server currently only exports via gRPC. The RPC exporter endpoint defaults tohttp://localhost:4317
. Endpoints can also be configured separately for metrics and (logs + traces) viaOTEL_EXPORTER_OTLP_METRICS_ENDPOINT
andOTEL_EXPORTER_OTLP_TRACES_ENDPOINT
.OTEL_METRIC_EXPORT_INTERVAL
- this defaults to 60000 (60 seconds), but I set it to 10000 or 15000 (10 or 15 seconds). I primarily export metrics to Mimir/Prometheus/Grafana where I want to evaluate or graph the metrics rate on one-minute intervals. The rule of thumb for Prometheus-compatible tooling seems to be an evaluation range >= 4x the scrape interval.
I have also added the options to enable or disable stdout/stderr logging, file logging, (logs + traces) exporting, and metrics exporting:
FILE_TRACES_EXPORTER_ENABLED
: defaultfalse
STD_STREAM_LOGS_EXPORTER_ENABLED
: defaultfalse
STD_STREAM_TRACES_EXPORTER_ENABLED
: defaultfalse
OTEL_COLLECTOR_LOGS_EXPORTER_ENABLED
: defaulttrue
OTEL_COLLECTOR_TRACES_EXPORTER_ENABLED
: defaulttrue
OTEL_COLLECTOR_METRICS_EXPORTER_ENABLED
: defaulttrue
Environment variable names and defaults subject to change.
File and stdout/stderr exporters are still OTEL-structured logs and traces, just not using the OTEL Rust exporter SDKs which are focused on exports over the wire to collectors.
In the development
directory, bring up the echo server with:
docker-compose up -d echo-server # add the --build option to include any local changes
The Docker Compose configuration binds the container port to the host network's port 8080.
Hit either the vanilla echo endpoint at localhost:8080
or the json echo endpoint at localhost:8080/json
:
curl -i -X GET localhost:8080/api/v0/echo -d 'hello world'
curl -i -X GET --header "content-type: application/json" localhost:8080/api/v0/echo/json -d '{"hello": "world"}'
Only use one of the two collectors for the telemetry data, either the OTEL Collector or Grafana Agent:
docker-compose up -d otel-collector
# or
docker-compose up -d grafana-agent
For now, the pre-set-up configuration options are more limited for the Grafana Agent.
The Docker Compose setup uses Grafana's Mimir metrics database in monolithic mode to ingest metrics, and Grafana (the visualization application itself) to query and graph the data.
Querying may also be done via HTTP calls to Mimir's Prometheus query endpoints.
docker compose up -d mimir grafana
Access the Grafana UI at localhost:3000/explore
and view the metrics with a PromQL query:
rate(http_server_request_duration_count[1m])
The easiest way to view traces is with the Jaeger all-in-one docker image. Jaeger added support for OpenTelemetry-formatted traces in v1.35
docker run -d --name jaeger \
-e COLLECTOR_OTLP_ENABLED=true \
-p 16686:16686 \
-p 4317:4317 \
-p 4318:4318 \
jaegertracing/all-in-one:1.35
Then access the Jaeger UI at http://localhost:16686. The echo server service will appear once traces have been produced. Make individual requests to the server or use the load gen scripts and container to create activity on the server.
Trace spans do not link together much at this point - I believe this is due to the lack of tracing support thus far in Hyper.
Compiling requires the protobuf compiler packages, usually called protobuf-devel
or similar in Linux repos