Tracing for debugging and performance management #1304

kayakr · 2019-10-11T19:55:40Z

As a developer seeking to debug Islandora 8, or a system administrator wanting to understand performance, I want to have easy access to logs across all of the components involved.

It is currently very challenging to diagnose issues such as failed derivative generation in Islandora 8, due to the numerous components involved, especially if some requests succeed and others fail, e.g. due to size, or format. Easy access to logs and highlighting errors can assist rapid diagnosis.

An example of desirable observability is the UI available from Jaeger, now part of the CNCF OpenTracing project https://opentracing.io/. The screenshots below are from Jaeger log aggregation from the demo HotROD application. They show the inferred dependency graph, the trace data, and drilling into a particular span to see the SQL query that was executed.

AFAICT tracing requires each component to be monitored to be able to accept or pass through a trace context. Tags can be added to provide global context, e.g. user or role, so that a derivative operation can be matched with a specific user.

jaeger-dag shows the components involved in a trace:

jaeger-trace shows the various services involved in a trace, their timing, highlighted errors, etc.:

jaeger-span-detail shows the detailed data for a given span, including arbitrary name-value pairs, the actual SQL command, etc.