Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Partial/Scheduled Rebalancing #2236

Open
imans777 opened this issue Jan 4, 2025 · 0 comments
Open

Partial/Scheduled Rebalancing #2236

imans777 opened this issue Jan 4, 2025 · 0 comments

Comments

@imans777
Copy link

imans777 commented Jan 4, 2025

Hi,
Currently when we call rebalancing of any kind, it starts instantly and continues until it finishes.
But in reality, clusters have some times which are at their peak load and some times at their lowest.
I think this is a very useful feature to do rebalancing when the load is at its lowest, so it affects the cluster/producer/consumers the least.
This can be achieved in two methods: partial rebalancing and scheduled rebalancing.

  • In former, when we call rebalancing, we can define some thresholds that when the load is at its lowest, it does a batch of rebalancing. This can also be done automatically in a way that CC looks at historical data, e.g. when we define a 1 week of windows, and finds times with lowest peak and then does its job.
  • In latter, e.g. we want to do a rebalancing at night because we know the load is lower, but it might be a crucial cluster that we're not allowed to have access during midnights, so we can schedule a time to start rebalancing and it doesn't start instantly.

Though I know that it's possible to stop execution to mimic partial rebalancing, and write a job to curl CC at our desired time to mimic the scheduling method, I believe these two features makes CC much more desirable and standalone.
Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant