Uploaded image for project: 'Apache Helix'
  1. Apache Helix
  2. HELIX-672

Improve partition distribution strategy for small clusters

    Details

    • Type: Task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.8.x
    • Component/s: None
    • Labels:
      None

      Description

      Current, CRUSH (also multi-round CRUSH) strategy provides even partition distribution based on statistically randomization. All CRUSH based strategy works well when the cluster is reasonably large.
      However, in some of our small production clusters, we find the distribution is not good as expected. The difference between heavy load node and idle node could be 30% in our case. Given we assign resource according to the max load, we are wasting 30% of resource on the idle nodes. And it could be worse in other clusters.
      We need to find a new algorithm or improve CRUSH for better evenness for small clusters.
      Note that, meanwhile, we should keep the good features such as minimum movements, deterministic calculating result.

        Attachments

          Activity

            People

            • Assignee:
              jiajunwang Jiajun Wang
              Reporter:
              jiajunwang Jiajun Wang
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: