Multiple consumers to support rabbitmq-sharding

[michaelklishin]

rabbitmq-sharding is an opinionated plugin which uses multiple "physical" queues behind one logical queue, taking advantage of more CPU cores.

This sounds like a great fit for the Logstash use case. However, to support it, the input needs to use multiple consumers.

This can be an option or an alternative input. Would there be any interest in this?

[andrewvc]

@michaelklishin I like this idea. Would you be up to contribute a patch here?

[michaelklishin]

@andrewvc yes, in a few weeks.

[GlenRSmith]

I'm quite interested in seeing improvements in throughput where RabbitMQ and Logstash are both in the data pipeline. (Evidence here: logstash-plugins/logstash-output-rabbitmq#25)

Is the proposal here to add configuration options to the ls-in-rmq plugin which would only work against an exchange that has the linked plugin installed?

[andrewvc]

@michaelklishin thinking through this I think that this should be a separate plugin if it's tightly bound to rabbitmq sharding. That being said if it's as simple as patching this to support multiple input queues (so you could pass an array of queues as an argument) then that would be a good patch to take.

[lsdev14]

Hey @michaelklishin, @andrewvc and @GlenRSmith any news about this topic?
Has any task started about it?

[andrewvc]

@lsdev14 I no longer work on logstash, but @jsvd may be able to help here.