Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Autoscale kinsumer #1190

Closed
bt-justtrack opened this issue Jan 6, 2025 · 0 comments · Fixed by #1192
Closed

Autoscale kinsumer #1190

bt-justtrack opened this issue Jan 6, 2025 · 0 comments · Fixed by #1192

Comments

@bt-justtrack
Copy link
Contributor

bt-justtrack commented Jan 6, 2025

Currently when scaling kinesis streams we need to redeploy eg the terraform code so we get a 1on1 mapping of consumer to shards (which is nice for load distribution purposes). This is tedious however, because you need to know which kinsumers are consuming the to be scaled stream.
It would be much nicer if the kinsumers would automatically be able to adjust to the number of shards. This should also not create any problems as eg terraform modules should also pull the amount of shards as data to scale on (you might only see some drift from the tf state which should consolidate itself on the next apply).

We want to do this with a dedicated module featuring leader election. The leader will then query the current amount of shards and directly set the ecs desired count to that amount.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant