Centrifuge Library Update 0.21.1 -> 0.32.2 Issues

[jherman]

Going FROM:
github.com/centrifugal/centrifuge v0.21.1
github.com/centrifugal/centrifuge-go v0.8.2

TO:
github.com/centrifugal/centrifuge v0.32.2
github.com/centrifugal/centrifuge-go v0.10.2

Redis and Sentinel Versions:
docker.io/bitnami/redis:7.0.12-debian-11-r19
docker.io/bitnami/redis-sentinel:7.0.12-debian-11-r18

I've done all the required refactoring to make things communicate on my local dev box using Memurai. Everything there works great. When I publish to k8s cluster, it typically uses our redis-sentinel helm chart app.

Our previous code for building shard configs contained only the following information:

redisConf := centrifuge.RedisShardConfig{
  SentinelAddresses:  sentinelURLs
  SentinelMasterName: "mymaster",
  ConnectTimeout:     3 * time.Second,
  IOTimeout:          3 * time.Second,
  Password:           "mypassword"
}

This worked fine and we were able to connect and use redis sentinel. Since upgrading centrifuge libraries, we've been getting spammed the following errors:

2024-07-29T13:47:54.324319987-04:00 {"severity":"info","time":1722275274,"message":"unknown control command: map[command:&controlpb.Command{state:impl.MessageState{NoUnkeyedLiterals:pragma.NoUnkeyedLiterals{}, DoNotCompare:pragma.DoNotCompare{}, DoNotCopy:pragma.DoNotCopy{}, atomicMessageInfo:(*impl.MessageInfo)(nil)}, sizeCache:0, unknownFields:[]uint8{0x1a, 0x51, 0xa, 0x24, 0x31, 0x31, 0x39, 0x38, 0x63, 0x30, 0x32, 0x39, 0x2d, 0x30, 0x36, 0x35, 0x33, 0x2d, 0x34, 0x39, 0x65, 0x32, 0x2d, 0x62, 0x32, 0x35, 0x30, 0x2d, 0x36, 0x33, 0x63, 0x36, 0x63, 0x30, 0x65, 0x65, 0x32, 0x37, 0x65, 0x30, 0x12, 0x25, 0x77, 0x73, 0x2d, 0x62, 0x72, 0x6f, 0x6b, 0x65, 0x72, 0x2d, 0x64, 0x65, 0x70, 0x6c, 0x6f, 0x79, 0x6d, 0x65, 0x6e, 0x74, 0x2d, 0x35, 0x63, 0x62, 0x36, 0x36, 0x39, 0x36, 0x63, 0x63, 0x66, 0x2d, 0x74, 0x37, 0x32, 0x6b, 0x78, 0x38, 0x9f, 0x94, 0x10}, Uid:\"1198c029-0653-49e2-b250-63c6c0ee27e0\", Node:(*controlpb.Node)(nil), Unsubscribe:(*controlpb.Unsubscribe)(nil), Disconnect:(*controlpb.Disconnect)(nil), Shutdown:(*controlpb.Shutdown)(nil), SurveyRequest:(*controlpb.SurveyRequest)(nil), SurveyResponse:(*controlpb.SurveyResponse)(nil), Subscribe:(*controlpb.Subscribe)(nil), Notification:(*controlpb.Notification)(nil), Refresh:(*controlpb.Refresh)(nil)}]"}
2024-07-29T13:47:54.324382663-04:00 {"severity":"info","time":1722275274,"message":"unknown control command: map[command:&controlpb.Command{state:impl.MessageState{NoUnkeyedLiterals:pragma.NoUnkeyedLiterals{}, DoNotCompare:pragma.DoNotCompare{}, DoNotCopy:pragma.DoNotCopy{}, atomicMessageInfo:(*impl.MessageInfo)(nil)}, sizeCache:0, unknownFields:[]uint8{0x1a, 0x51, 0xa, 0x24, 0x31, 0x31, 0x39, 0x38, 0x63, 0x30, 0x32, 0x39, 0x2d, 0x30, 0x36, 0x35, 0x33, 0x2d, 0x34, 0x39, 0x65, 0x32, 0x2d, 0x62, 0x32, 0x35, 0x30, 0x2d, 0x36, 0x33, 0x63, 0x36, 0x63, 0x30, 0x65, 0x65, 0x32, 0x37, 0x65, 0x30, 0x12, 0x25, 0x77, 0x73, 0x2d, 0x62, 0x72, 0x6f, 0x6b, 0x65, 0x72, 0x2d, 0x64, 0x65, 0x70, 0x6c, 0x6f, 0x79, 0x6d, 0x65, 0x6e, 0x74, 0x2d, 0
x35, 0x63, 0x62, 0x36, 0x36, 0x39, 0x36, 0x63, 0x63, 0x66, 0x2d, 0x74, 0x37, 0x32, 0x6b, 0x78, 0x38, 0x9f, 0x94, 0x10}, Uid:\"1198c029-0653-49e2-b250-63c6c0ee27e0\", Node:(*controlpb.Node)(nil), Unsubscribe:(*controlpb.Unsubscribe)(nil), Disconnect:(*controlpb.Disconnect)(nil), Shutdown:(*controlpb.Shutdown)(nil), SurveyRequest:(*controlpb.SurveyRequest)(nil), SurveyResponse:(*controlpb.SurveyResponse)(nil), Subscribe:(*controlpb.Subscribe)(nil), Notification:(*controlpb.Notification)(nil), Refresh:(*controlpb.Refresh)(nil)}]"}

I've traced this back to an internal errors within the node.go file:

centrifuge/node.go

Line 683 in 70fd357

n.logger.log(newLogEntry(LogLevelError, "unknown control command", map[string]any{"command": fmt.Sprintf("%#v", cmd)}))

Sounds like the redis/sentinel is issuing a command not recognized?

I've tried playing around with different shard configs. However I've come across different issues. I noticed in particular that our previous developer used the "Password" property vs "SentinelPassword". I attempted to swap that as:

redisConf := centrifuge.RedisShardConfig{
  SentinelAddresses:  sentinelURLs
  SentinelMasterName: "mymaster",
  ConnectTimeout:     3 * time.Second,
  IOTimeout:          3 * time.Second,
  SentinelPassword:   "mypassword"
}

The error I now receive is when I attempt to build a new RedisShardConfig:

NOAUTH HELLO must be called with the client already authenticated, otherwise the HELLO AUTH <user> <pass> option can be used to authenticate the client and select the RESP protocol version at the same time

I've Googled around and I can't come up with much of a solution. I feel I've exhausted a bunch of resources, and so here I am. Any advice any of you all might have?

[FZambia]

@jherman hello!

The reason you get spammy unknown control command messages is because you are utilizing one Redis for different versions of Centrifuge/Centrifugo. It's actually may be a signal that most probably you are re-using the same Redis with the same Redis prefix across different applications.

The format of internal protocol changed at some point in v0.29.0 making it impossible to run Centrifuge nodes of different versions (< v0.29.0 and >=v0.29.0) in one cluster.

The advice is simple - you should not mix Centrifuge/Centrifugo clusters of different versions with the same Redis. It's usually a mistake on its own (if Redis prefix is not configured correctly), but in this case since internal protocol change one version of Centrifuge/Centrifugo setup collapses with another one.

[jherman]

Hello @FZambia - you nailed it on the head. This was driving me nuts for a week. Currently in our dev environment we run multiple branches and thus have multiple Centrifuge nodes. I stopped the other branch, and just like you said, it fixed the problem. Thank you so much for the help!