HBase
  1. HBase
  2. HBASE-1676

load balancing on a large cluster doesn't work very well

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.0
    • Fix Version/s: 0.90.0
    • Component/s: None
    • Labels:
      None

      Description

      I have 200 regions per server in my cluster. When I take one down, then bring it back up, I only end up with about 30 regions assigned to it. This is highly reproducible.

      Furthermore, the assignment algorithm will do silly things like assign all regions of a table to one regionserver - even if it's the best server. We need to be smarter about optimizing how regions are assigned if we want to achieve maximal performance on our clusters.

        Issue Links

          Activity

          ryan rawson created issue -
          Hide
          stack added a comment -

          Lets move the assignment code to its own class to facilitate testing alogrithm in isolation. Then write one million unit tests for it.

          Show
          stack added a comment - Lets move the assignment code to its own class to facilitate testing alogrithm in isolation. Then write one million unit tests for it.
          Hide
          Bryan Duxbury added a comment -

          +1 to that idea.

          Show
          Bryan Duxbury added a comment - +1 to that idea.
          Hide
          stack added a comment -

          Too big to fix in a point release. Moving to 0.21.

          Show
          stack added a comment - Too big to fix in a point release. Moving to 0.21.
          stack made changes -
          Field Original Value New Value
          Fix Version/s 0.21.0 [ 12313607 ]
          Fix Version/s 0.20.1 [ 12313606 ]
          stack made changes -
          Link This issue is part of HBASE-1816 [ HBASE-1816 ]
          Hide
          stack added a comment -

          Master Rewrite should make it so assignment is testable outside of region context.

          Show
          stack added a comment - Master Rewrite should make it so assignment is testable outside of region context.
          stack made changes -
          Link This issue is part of HBASE-2480 [ HBASE-2480 ]
          stack made changes -
          Link This issue is blocked by HBASE-2699 [ HBASE-2699 ]
          Hide
          Jonathan Gray added a comment -

          There are standalone tests of the new load balancer! This is open against 0.90. Do we want to close it or what exactly is the scope of this jira? IMO this is done, other stuff can be handled in "smart" load balancing and other open jiras.

          Show
          Jonathan Gray added a comment - There are standalone tests of the new load balancer! This is open against 0.90. Do we want to close it or what exactly is the scope of this jira? IMO this is done, other stuff can be handled in "smart" load balancing and other open jiras.
          Hide
          stack added a comment -

          Resolving as done as part of master rewrite issue. Open new issue if the new load balancer committed there is unsatisfactory.

          Show
          stack added a comment - Resolving as done as part of master rewrite issue. Open new issue if the new load balancer committed there is unsatisfactory.
          stack made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Resolved Resolved
          407d 7h 45m 1 stack 01/Sep/10 06:42

            People

            • Assignee:
              Unassigned
              Reporter:
              ryan rawson
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development