You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently when scaling kinesis streams we need to redeploy eg the terraform code so we get a 1on1 mapping of consumer to shards (which is nice for load distribution purposes). This is tedious however, because you need to know which kinsumers are consuming the to be scaled stream.
It would be much nicer if the kinsumers would automatically be able to adjust to the number of shards. This should also not create any problems as eg terraform modules should also pull the amount of shards as data to scale on (you might only see some drift from the tf state which should consolidate itself on the next apply).
We want to do this with a dedicated module featuring leader election. The leader will then query the current amount of shards and directly set the ecs desired count to that amount.
The text was updated successfully, but these errors were encountered:
Currently when scaling kinesis streams we need to redeploy eg the terraform code so we get a 1on1 mapping of consumer to shards (which is nice for load distribution purposes). This is tedious however, because you need to know which kinsumers are consuming the to be scaled stream.
It would be much nicer if the kinsumers would automatically be able to adjust to the number of shards. This should also not create any problems as eg terraform modules should also pull the amount of shards as data to scale on (you might only see some drift from the tf state which should consolidate itself on the next apply).
We want to do this with a dedicated module featuring leader election. The leader will then query the current amount of shards and directly set the ecs desired count to that amount.
The text was updated successfully, but these errors were encountered: