Startup order in Kubernetes

[DNCoelho]

Say you have a working clipper instance deployed in Kubernetes which uses an external persistent Redis instance provided via a statefulset. If for some reason the system stops working and a restart is needed, there is a possibility that the QueryFronend will startup before the Redis instance, which will cause the system to malfunction as it does not detect one of its components. Is there any way of preventing this behaviour beyond altering the QueryFrontend definition in order to include prevention mechanisms, such as init containers?

[withsmilo]

@DNCoelho : We solved this problem by installing Redis + Sentinel + HAProxy on the external servers to increase system stability for our production environment.

[DNCoelho]

@withsmilo
Thank you for your reply!
I see what you mean, by increasing stability there should never occur a case where my initial situation may occur. However, it may still happen in which case there would exist a chance of the restart order being an erroneous one. In that case, is there any way to prevent it? Or do you think the possibility of it happening is so remote that the situation may not even warrant an explanation?

[simon-mo]

Hmmm. I can see two alternatives for clipper:

1. Implement init-containers to block query frontend until redis is healthy
2. Have query frontend explicitly block (with exponential backoff and retries count configurable) until redis is ready.

2 seems easier and more native.

[DNCoelho]

@simon-mo
I agree that 2 seems like a better option. In the future, I'll try to change the frontend definition to accommodate this behavior. It may be a good idea to incorporate some sort of similar behavior in the official image though, has this is an issue that anyone who provides an external Redis instance will encounter.