x-expires for streams

[0UserName]

Question

Are x-expires support planned for stream queues?

---

Use case

The system processes the incoming messages, which in turn are subdivided into subtypes. In addition to subtypes, each message has a certain identifier that allows to determine the belonging of messages to a specific event/entity.

For some of the subtypes of each event/entity (identifier), it is required to create a stream in order to be able to read messages again on demand (user request).

Streams storing the events/entities data should have a lifetime of N days (x-expires), since during this period the events/objects will expire and storing the data no longer makes sense.

Streams have support for x-max-age, which solves the problem of storing outdated data, but the streams themselves continue to function as before. Since there can be many events/entities (hundreds, thousands, etc), this will lead to garbage in the topology.

It would be convenient to be able to automatically delete streams.

---

What was tested

RabbitMQ version: 3.9.6

Using x-expires/ expires as an argument passed to declare method/policy does not work for streams, but works for classic/quorum queues

[michaelklishin]

See Stream Retention Policies. Streams are not queues, and even though in some cases clients can use them as if they were, not every feature makes sense for every type, even quorum and classic queues have differences.

[0UserName]

See Stream Retention Policies.

As I understand it from the description (I mentioned it in the question), these options apply to data. In other words, mass deletion of outdated streams is impossible.

---

Streams are not queues

The semantics are still similar. Some of the scenarios are identical to classic queues from the point of view of the end client.

---

not every feature makes sense for every type

Do you think that the described scenario does not make sense?

[michaelklishin]

Mass deletion of durable queue types and streams isn't particularly popular. No one on our team or outside has asked for stream TTL before.

@kjnilsson do you have an opinion on stream TTL?

[michaelklishin]

In a Slack discussion with several members of the team, there was little interest in making data types that are replicated and designed to focus on safety (streams is one of them) self-destruct. Programmatically deleting streams is already possible, so this is no a priority.

That said, RabbitMQ is open source software and a reasonable implementation of stream TTL would be considered.