Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Investigate use of split_partition_queue to simplify StreamProcessor #413

Open
untitaker opened this issue Dec 18, 2024 · 0 comments
Open

Comments

@untitaker
Copy link
Member

getsentry/taskbroker is using split_partition_queue, and that allows them to poll the consumer separately from pulling messages out of it.

Right now we pause the consumer on backpressure to not receive messages. If we used split_partition_queue, that would be obsolete and handling backpressure in arroyo would become simpler.

It seems however that this is not an option for python, as confluent-kafka doesn't have split_partition_queue. Might be a Rust-only thing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant