Add built-in health check functionality for systems like Kubernetes

[jbdietrich]

Description

We want to integrate the racecar consumer health-check readiness probe into racecar itself. As a result, the health check functionality will be part of racecar so that racecar users do not need to implement the logic as well as the bash scripts by themselves. In another word, to check the heathiness of racecar, the racecar users only need to include the built-in probe in there K8s manifest.

The readiness probe touches the file to update the creation timestamp in every iteration of the racecar main loop. When the racecar is down, the timestamp gets expired so that the readiness probe which checks the timestamp fails.

We also need to update the handbook of racecar to introduce how to make use of the integrated readiness probe. The probe is a bash script and two env vars are needed.

[edit: description provided by @BingkunWu]

[meqif]

@jbdietrich This is a nice improvement, thank you!

However, I found that due to a bug in rdkafka, sometimes the consumer stops processing one or more partitions. That sort of issue is not detectable through a consumer-wide liveness check because the consumer keeps making progress in some but not all assigned partitions.

What worked for me was tracking the liveness of each assigned partition by 1. refreshing files that represent the liveness of each assigned partition and 2. registering a rebalance listener that manages (deletes and/or creates) those partition liveness files. This solution has already saved my bacon at least once. :)

I can open a draft pull request with that strategy, if you think that'd be valuable.

[jbdietrich]

Thanks for your response @meqif!

I can open a draft pull request with that strategy, if you think that'd be valuable.

That sounds great, please do!

[meqif]

Hi @jbdietrich, sorry for the delay! I've opened a draft PR (#309) documenting how a liveness probe as I described might be implemented.

[jbdietrich]

@meqif no problem, thanks for putting up the PR. I am not sure whether we will get to it before the holidays unfortunately. If not, I'll have the team review early in 2023.

[meqif]

@jbdietrich That's okay, there's no rush on my end. I'll also be leaving for the holidays soon. :)

After all, that PR is simply adding documentation for an approach we've been using (plus a small code change which would allow us to remove a monkey patch). We're not blocked in any way.

[jbdietrich]

Superseded by #319