Skip to content

Commit

Permalink
redis-ha: fix readiness probe for non-connected slaves
Browse files Browse the repository at this point in the history
When Redis is running as a slave, don't pass readiness probes until and
unless the replica is fully synchronised.

This resolves the issue where, when Redis is upgraded, the replica is
marked ready while still syncing from the master. This can lead to the
master pod being terminated and restarted with no slaves ready to take
over from it, leaving the Redis cluster failed until the original
primary comes back up again.

Signed-off-by: Chris Boot <[email protected]>
  • Loading branch information
bootc committed Aug 22, 2024
1 parent 408c54c commit 32f960c
Show file tree
Hide file tree
Showing 2 changed files with 32 additions and 3 deletions.
2 changes: 1 addition & 1 deletion charts/redis-ha/Chart.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ keywords:
- redis
- keyvalue
- database
version: 4.27.2
version: 4.27.3
appVersion: 7.2.4
description: This Helm chart provides a highly available Redis implementation with a master/slave configuration and uses Sentinel sidecars for failover management
icon: https://upload.wikimedia.org/wikipedia/en/thumb/6/6b/Redis_Logo.svg/1200px-Redis_Logo.svg.png
Expand Down
33 changes: 31 additions & 2 deletions charts/redis-ha/templates/_configs.tpl
Original file line number Diff line number Diff line change
Expand Up @@ -668,10 +668,39 @@
ping
)
if [ "$response" != "PONG" ] ; then
echo "$response"
echo "ping=$response"
exit 1
fi

response=$(
redis-cli \
{{- if .Values.auth }}
-a "${AUTH}" --no-auth-warning \
{{- end }}
-h localhost \
{{- if ne (int .Values.redis.port) 0 }}
-p {{ .Values.redis.port }} \
{{- else }}
-p {{ .Values.redis.tlsPort }} ${TLS_CLIENT_OPTION} \
{{- end}}
role
)
role=$( echo "$response" | sed "1!d" )
if [ "$role" = "master" ]; then
echo "role=$role"
exit 0
elif [ "$role" = "slave" ]; then
repl=$( echo "$response" | sed "4!d" )
echo "role=$role; repl=$repl"
if [ "$repl" = "connected" ]; then
exit 0
else
exit 1
fi
else
echo "role=$role"
exit 1
fi
echo "response=$response"
{{- end }}

{{- define "sentinel_liveness.sh" }}
Expand Down

0 comments on commit 32f960c

Please sign in to comment.