Rebalance in a Disk Aware Manner

[imans777]

Hi,
As far as I know and saw in code and also in experience, cruise control does inter-broker and intra-broker rebalancing in two different ways and it's not possible to merge them in a single rebalance execution.
Why this is very important?
An example scenario is that we rebalanced a cluster, then on one broker, there are couple of disks, and there are two replicas on each. Two replicas which are leader stay in a single disk, and the disk becomes bottleneck. We have to run a separate rebalancing with rebalance_disk=true to resolve this (though I'm not sure it handles this situation, it only has a disk distribution goal), and if it's not resolved, manual actions should be taken.
Also we don't want to have a multi-step rebalancing. We want the rebalancing to does all the jobs we asked in a single execution, and we expect the cluster to be rebalanced without any considerations.
All and all, this has become the main issue in our cluster which has lots of brokers with multiple disks each, and we don't have enough visibility on disk granularity and cc loses most of its promises to be an exclusive kafka management.
I believe this can be resolved in two steps:
1- rebalance_disk should become more mature, accepting more goals like i/o, etc (I created another issue for this #2232 )
2- rebalance_disk should become a goal, which can be run along with normal /rebalance and with a single execution.
Thanks.