Periodic message accumulation in redis

[mbanaszkiewicz-vonage]

My use case - periodic job distribution over multiple ECS app instances:

1. ~20k unique jobs
2. jobs created independently at various times (will filter out duplicates to keep old ones I guess via https://sonus21.github.io/rqueue/message-deduplication)
3. each job should be handled only once exactly every 5s, by one of my app instances until it is deleted (I suppose in RqueueListener, when some condition is met).

I wanted to experiment (see below demo repo with spring boot app), at app startup I'm creating a periodic message, scheduled to run every 5s, with constant message id. I've noticed that each 5s a new entry is created in redis: __rq::queue::{my-delay-queue}::MSG_ID::sch::1729507117728, with ttl of 30mins. I've tried to utilize message.durability.in-terminal-state=0 / job.enabled=false / job.durability (see application.yml) but it doesn't seem to do anything in that case.

• I'm concerned that this will be a redis memory issue after reaching target 20k messages, is there any other config option for this that I've missed?
• after looking at my app logs from message consumption, I've noticed that the times of consumption vary a little bit and are not always 5s (sometimes it's 3.5s, sometimes it's 6s), is this something I should expect?

Steps to reproduce:

• clone https://github.com/mbanaszkiewicz-vonage/rqueue-experiments
• run redis docker container (readme in repo)
• redis-cli: keys *

[sonus21]

Thanks for bringing this discussion.

• These keys are used to avoid duplicate message enqueue due to retry in any job.
• The simplest way to optimize the the memory usage is set the queue visibility timeout, in this case you can set 1 or 2 minutes. The TTL of the keys would be based on that, given you’re recurring job at 5 seconds interval, you can also try to set this number as low as 10 seconds, in such case the key would be stored only for 20 seconds.
• The message should be consumed at constant interval but some time it can change. This happens due to slow polling, if you want aggressive polling, you can reduce the polling interval at factory level.

[mbanaszkiewicz-vonage]

Thank you for the reply.

• using visibilityTimeout = "10000" in my RqueueListener indeed reduced the count of current keys
• I tried reducing the poling interval to improve scheduled consumptions timings, like in example

  rqueue/rqueue-spring-boot-reactive-example/src/main/java/com/github/sonus21/task/executor/RqueueReactiveApplication.java

  Line 36 in f04317b

  @Bean

  , but it doesn't seem to have any impact, I've tried 100 & 1000 millis: https://github.com/mbanaszkiewicz-vonage/rqueue-experiments/blob/6db181ee36d2c47791efd3fd2b2513bb7ea4fedc/src/main/java/com/example/Application.java#L12

2024-10-23T08:23:58.426545654Z calling enqueuePeriodic...
2024-10-23T08:24:04.506793614Z Handling message (id: MSG_ID, body: {"msg":""MSG_BODY"","name":"java.lang.String"})
2024-10-23T08:24:10.415605790Z Handling message (id: MSG_ID, body: {"msg":""MSG_BODY"","name":"java.lang.String"})
2024-10-23T08:24:14.458717738Z Handling message (id: MSG_ID, body: {"msg":""MSG_BODY"","name":"java.lang.String"})
2024-10-23T08:24:20.439327411Z Handling message (id: MSG_ID, body: {"msg":""MSG_BODY"","name":"java.lang.String"})
2024-10-23T08:24:24.446374419Z Handling message (id: MSG_ID, body: {"msg":""MSG_BODY"","name":"java.lang.String"})
2024-10-23T08:24:30.407433503Z Handling message (id: MSG_ID, body: {"msg":""MSG_BODY"","name":"java.lang.String"})
2024-10-23T08:24:34.405686303Z Handling message (id: MSG_ID, body: {"msg":""MSG_BODY"","name":"java.lang.String"})

[sonus21]

What’s your concurrency setting for this queue? When you’ve large number of messages you would also need higher concurrency?

can you try setting?

rqueue.scheduler.scheduled.message.time.interval to 100