[bug] internal server error on publish

[hetao29]

Describe the bug
on Publish message, some time error: internal server error ,and code is 100, and it recover after 1-3 minis later.
I have searched the issue, Like:
#643

I think the reason mayby is:

1. centrifugo config allow history
2. redis drop some key by allkey-lfu config

maxmemory 500M
maxmemory-policy allkeys-lfu

3. and publish the message and add it to history happen error.

thanks.

Versions

Centrifugo version is 5.4.5
Operating system is ubuntu 22.04 , docker centrifugo/centrifugo:v5.4.5

** config below:

{
	"api_key": "xx",
	"admin_password": "xx",
	"admin_secret": "xx",
	"engine": "redis",
	"redis_cluster_address":[
		"xxx"
	],
	"token_hmac_secret_key": "xx",
	"port":8000,
	"ping_interval":"20s",
	"pong_timeout":"5s",
	"redis_prefix": "centrifugov5.1",
	"uni_http_stream":true,
	"namespaces": [
		{
			"name": "aa",
			"presence": true,
			"join_leave":true,
			"history_size": 50,
			"history_ttl": "300s",
			"force_push_join_leave": true,
			"allow_subscribe_for_anonymous":true,
			"allow_subscribe_for_client":true
		}
	],
	"allowed_origins": [
		"*"
	]
}

[FZambia]

Hello @hetao29

I think logs should contain more specific error returned from Redis, not just internal error exposed to the outside.

Generally you can't use Centrifugo history with Redis LRU policies, because it may bring unexpected side effects to message offset logic upon key eviction.

Though I'd like to look at exact error that is happening in this particular case.

[hetao29]

Hello @hetao29

I think logs should contain more specific error returned from Redis, not just internal error exposed to the outside.

Generally you can't use Centrifugo history with Redis LRU policies, because it may bring unexpected side effects to message offset logic upon key eviction.

Though I'd like to look at exact error that is happening in this particular case.

It was a problem in the production environment. I took a rough look at the centrifugo and redis logs and found no obvious error logs; the initial estimate is that it is related to history;
Now the configuration file has deleted the history-related configuration;
But what I think is that if it is combined with redis lru, it may be better to have better fault tolerance;

[FZambia]

Would be nice to see the exact error. Will try to reproduce this myself at some point, and think whether it may be improved somehow. There is an idea to add an option to only use one key for the history instead of two. If possible, it will provide a bit worse characteristics for the recovery in some edge cases, but should be more friendly to cache eviction.