Major Description
There are a lot of user questions around how rebalance works, configuration etc.
Example from Gideon: I still think we need to make important improvements to the rebalancing documentation. One "big picture" item is to explain the resource manager's role in rebalancing activity (RM isn't mentioned in the doc's in this context).
We should also add more detail explaining how to optimize multi-threaded rebalancing.
What exactly happens with multi-threaded rebalancing, and what are the limits and/or consequences? For example, could too high a degree of parallelism defeat the rebalancing algorithm (given the original design targets reaching the right end-state by moving buckets one at-a-time)? Or, is the reverse true, and we might bet better final results with more threads ? Is there some guideline we can devise based on the number of nodes in the cluster, the number of cores per server, and the configured number of buckets for a PR? Is multi-threaded applied on each host individually, or are the extra threads only running on the rebalance "coordinator" node?
I wasn't able to explain any of the nuances of rebalancing based on our docs and cursory review of the Geode codebase (although for the latter I'm sure I could eventually . . .). This actually screams for a dedicated section on the Geode Wiki "Geode Internal Architecture" section . . . I would be happy to help write this I can get get started with answers to the above questions.