Telemetry instrumentation vs Health Checking

[julealgon]

I've been using these OTEL instrumentation libraries a lot recently in conjunction with the OTEL Collector and Datadog as the monitoring tool.

Over the past few days, I started to take a look into Health Checking in AspNetCore, through this page:

• Health checks in ASP.NET Core

After reading a bit on this, and then taking a look at the various health check-related packages I'm a bit conflicted: there seems to be quite a bit of conceptual overlap between telemetry/metrics instrumentation, and these health checking procedures.

This is not so much a question but a discussion. How does the team/community view this potential overlap? Which one will "win", or will we end up using both mechanisms at the same time in the future?

The way I'm seeing these health checking packages is that they look a lot like OpenTelemetry instrumentation packages to me. Usually, its one package per "technology", and it produces some sort of "metric" or a communication result that serves to signal that the service is ok.

The healthchecking system even includes the "publisher" abstraction: a hosted service that pushes health data from time to time to some receiver (again, very similar to me to what an OTEL exporter would do).

I also imagine one would want to see "application health" in the same tool that is used for monitoring logs/traces/metrics, at least in most situations, so there is a conceptual overlap in terms of exporting the data as well.

Should we even use both systems at the same time? Is there some guideline or recommendation in regards to the interaction between OTEL telemetry instrumentation/exporting vs this HealthCheck mechanism?

As it stands, the OTEL instrumentation can't be used for the same purposes that Health Checking is used, correct? For example, docker/Kubernetes can use the response from the HealthCheck endpoints to detect whether the application is ready to receive requests or if its in some sort of failure state for example. On the other hand, OTEL instrumentation provides metrics from time to time based on each instrumentation and batching settings etc, and doesn't have some "real-time" way of checking the health of the metered components.

Does OTEL itself define/discuss anything health-related? Would it ever make sense to combine these concepts into one, and have health be part of OTEL and be able to be exported to monitoring tools in a similar way?

How are you guys currently handling this? Do you have both things in place, or do you only use OTEL instrumentation?

Thoughts?