Better handling of master failover

[laurivosandi]

It looks like most applications do have Redis reconnect support if Redis operation results in error such as ReplyError: READONLY You can't write against a read only replica. In fact the ioredis docs specifically mention this usecase.

This is related to #149 (comment)

To better support dumb clients I would advise implementing TCP disconnect when Dragonfly pod loses master role to ensure clients reconnect to the new Dragonfly master pod

[Pothulapati]

Hmm, This makes sense but

To better support dumb clients I would advise implementing TCP disconnect when Dragonfly pod loses master role to ensure clients reconnect to the new Dragonfly master pod

How can we do this? Considering we don't have a proxy to do different replies. It's upto the older pod to return READONLY (which they will once they are configured as a replica). wdyt? 🤔

[laurivosandi]

I think it would be fine to issue TCP connection reset for every open connection if pod loses write master status. You can make this behaviour optional via command line flag or something like that

[nmass-betpawa]

Any updates regarding this problem?

[Pothulapati]

I think it would be fine to issue TCP connection reset for every open connection if pod loses write master status. You can make this behaviour optional via command line flag or something like that

Functionality to kill client connections currently does not exist in Dragonfly (See dragonflydb/dragonfly#2178). and there is no common Kubernetes abstraciton/functionality to do this. :/

cc: @romange

[jonathon2nd]

Hello!

I we are trying to do some tests with failover and wondering the best way to do this. Don't want to delete the master pod, as that always works.

Want to issue a failover, but

127.0.0.1:6379> failover
(error) ERR unknown command `FAILOVER`

And when I try manually

127.0.0.1:6379> REPLICAOF NO ONE
OK

on replica works, but then when I try to get the old master to follow new one this happens.

127.0.0.1:6379> REPLICAOF 10.42.90.34 6379
(error) ERR could not greet master Bad message

Any ideas?

[jonathon2nd]

found reference to a command called REPLTAKEOVER, but does not seem like I can get it to do anything, help is also empty

10.43.107.71:6379> REPLTAKEOVER "10.42.206.142" "6379"
(error) ERR value is not an integer or out of range
10.43.107.71:6379> REPLTAKEOVER
(error) ERR wrong number of arguments for 'repltakeover' command
10.43.107.71:6379> help REPLTAKEOVER