Enable tracing in Redis clients

[sevein]

This pull request enables tracing instrumentation in our Redis clients using the redisotel package.

TraceQL query examples

Find watcher triggers

This is likely the most interesting outcome of this work. Now watchers start a root span (watcher.poll) representing the interaction with Redis using the blmove command to wait until the notification is delivered by MinIO. In Grafana, search for:

{name=~"watcher.poll"}

The gantt chart shows that we remove the notification from the queue using lrem command, but only once we've successfully started the workflow instance. The workflow continues running and recording spans as part of the same trace, including the entire workflow execution, activities, etc...

Find creation of WebSocket auth tickets

We can see how authentication tickets are stored in Redis:

{name="api" && .http.route="/package/monitor"}

Find application publishing events

We can also find spans related to the operation of publishing events into the event service, e.g.:

{name="publish"}

Find Redis traces

A more general approach:

{.db.system="redis"}

[codecov]

Codecov Report

Attention: Patch coverage is 45.28302% with 29 lines in your changes are missing coverage. Please review.

Project coverage is 47.34%. Comparing base (7e99584) to head (0fb0243).

❗ Current head 0fb0243 differs from pull request most recent head 39e0299. Consider uploading reports for the commit 39e0299 to get more accurate results

Files	Patch %	Lines
main.go	0.00%	15 Missing ⚠️
internal/watcher/minio.go	71.42%	2 Missing and 2 partials ⚠️
cmd/enduro-a3m-worker/main.go	0.00%	2 Missing ⚠️
cmd/enduro-am-worker/main.go	0.00%	2 Missing ⚠️
internal/api/auth/ticket_store.go	77.77%	1 Missing and 1 partial ⚠️
internal/event/redis.go	77.77%	1 Missing and 1 partial ⚠️
internal/watcher/watcher.go	0.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #893      +/-   ##
==========================================
+ Coverage   47.27%   47.34%   +0.07%     
==========================================
  Files          95       95              
  Lines        5172     5206      +34     
==========================================
+ Hits         2445     2465      +20     
- Misses       2494     2505      +11     
- Partials      233      236       +3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[djjuhasz]

Cool! 😎