Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.95.2
    • Fix Version/s: 0.95.2
    • Component/s: master
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Added a new StochasticLoadBalancer that when enabled will perform a randomized search for the optimal cluster balance. The new balancer takes into account data locality, storefile size, memstore size, and the evenness of tables over region servers when trying potential new cluster states.

      To enable the new balancer set hbase.master.loadbalancer.class to org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer . It is also recommended to set hbase.master.loadbalance.bytable to false . Lots of different configuration options can be tuned to prioritize costs differently. Explanations of all of the configuration options are available on the JavaDoc for StochasticLoadBalancer.

      StochasticLoadBalancer is the default in 0.96.0
      Show
      Added a new StochasticLoadBalancer that when enabled will perform a randomized search for the optimal cluster balance. The new balancer takes into account data locality, storefile size, memstore size, and the evenness of tables over region servers when trying potential new cluster states. To enable the new balancer set hbase.master.loadbalancer.class to org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer . It is also recommended to set hbase.master.loadbalance.bytable to false . Lots of different configuration options can be tuned to prioritize costs differently. Explanations of all of the configuration options are available on the JavaDoc for StochasticLoadBalancer. StochasticLoadBalancer is the default in 0.96.0

      Description

      Now that balancers are pluggable we should give some options.

      1. HBASE-5959-9.patch
        152 kB
        Elliott Clark
      2. HBASE-5959-8.patch
        150 kB
        Elliott Clark
      3. HBASE-5959-7.patch
        149 kB
        Elliott Clark
      4. HBASE-5959-6.patch
        146 kB
        Elliott Clark
      5. HBASE-5959-3.patch
        153 kB
        Elliott Clark
      6. HBASE-5959-2.patch
        152 kB
        Elliott Clark
      7. HBASE-5959-14.patch
        164 kB
        Elliott Clark
      8. HBASE-5959-13.patch
        164 kB
        Elliott Clark
      9. HBASE-5959-12.patch
        162 kB
        Elliott Clark
      10. HBASE-5959-11.patch
        162 kB
        Elliott Clark
      11. HBASE-5959-1.patch
        142 kB
        Elliott Clark
      12. HBASE-5959-0.patch
        128 kB
        Elliott Clark
      13. ASF.LICENSE.NOT.GRANTED--HBASE-5959.D3189.7.patch
        115 kB
        Phabricator
      14. ASF.LICENSE.NOT.GRANTED--HBASE-5959.D3189.6.patch
        115 kB
        Phabricator
      15. ASF.LICENSE.NOT.GRANTED--HBASE-5959.D3189.5.patch
        115 kB
        Phabricator
      16. ASF.LICENSE.NOT.GRANTED--HBASE-5959.D3189.4.patch
        114 kB
        Phabricator
      17. ASF.LICENSE.NOT.GRANTED--HBASE-5959.D3189.3.patch
        111 kB
        Phabricator
      18. ASF.LICENSE.NOT.GRANTED--HBASE-5959.D3189.2.patch
        111 kB
        Phabricator
      19. ASF.LICENSE.NOT.GRANTED--HBASE-5959.D3189.1.patch
        111 kB
        Phabricator

        Activity

        Hide
        Elliott Clark added a comment -

        First stab at it. So far there's no tests and I haven't done a whole lot of testing or re-factoring.

        Needs a whole lot more cost functions.

        Show
        Elliott Clark added a comment - First stab at it. So far there's no tests and I haven't done a whole lot of testing or re-factoring. Needs a whole lot more cost functions.
        Hide
        Ted Yu added a comment -

        For this feature, design document showing the origin of Stochastic approach would help reviewers understand better.

        +class BalanceInfo {
        +
        +  private final int nextRegionForUnload;
        

        Please what the integer represents.

        +public abstract class BaseLoadBalancer implements LoadBalancer {
        

        Please add javadoc for the above class.

        +  protected HTableDescriptor getTableDescriptor(byte[] tableName) throws IOException {
        

        Why would the above method reside in BaseLoadBalancer ?
        StochasticLoadBalancer, it starts to get interesting.

        +    // Perform a stocastic walk to see if we can get a
        

        Please finish the above comment.

        +        if (lRegion != null) {
        +          rightRegionList.add(lRegion);
        +        }
        +
        +        double newCost = computerCost(initialRegionMapping, clusterState);
        

        computerCost() should be named computeCost(). How would the above call utilize rightRegionList ?

        +        if (newCost < currentCost || RANDOM.nextFloat() < 0.01) {
        +          currentCost = newCost;
        +        } else {
        

        What if newCost is much worse than currentCost but we choose it due to RANDOM.nextFloat() < 0.01 ?

        +    double skewCost = computeSkewLoadCost(clusterState);
        +    return moveCost + (100 * skewCost);
        

        Why would skewCost carry such a big weight ?

        Please add test for StochasticLoadBalancer.

        Show
        Ted Yu added a comment - For this feature, design document showing the origin of Stochastic approach would help reviewers understand better. +class BalanceInfo { + + private final int nextRegionForUnload; Please what the integer represents. + public abstract class BaseLoadBalancer implements LoadBalancer { Please add javadoc for the above class. + protected HTableDescriptor getTableDescriptor( byte [] tableName) throws IOException { Why would the above method reside in BaseLoadBalancer ? StochasticLoadBalancer, it starts to get interesting. + // Perform a stocastic walk to see if we can get a Please finish the above comment. + if (lRegion != null ) { + rightRegionList.add(lRegion); + } + + double newCost = computerCost(initialRegionMapping, clusterState); computerCost() should be named computeCost(). How would the above call utilize rightRegionList ? + if (newCost < currentCost || RANDOM.nextFloat() < 0.01) { + currentCost = newCost; + } else { What if newCost is much worse than currentCost but we choose it due to RANDOM.nextFloat() < 0.01 ? + double skewCost = computeSkewLoadCost(clusterState); + return moveCost + (100 * skewCost); Why would skewCost carry such a big weight ? Please add test for StochasticLoadBalancer.
        Hide
        Elliott Clark added a comment -

        Better version with factored tests.

        It still needs some more documentation and I'm hoping to add more cost functions. Though I might leave that for another issue.

        Show
        Elliott Clark added a comment - Better version with factored tests. It still needs some more documentation and I'm hoping to add more cost functions. Though I might leave that for another issue.
        Hide
        Elliott Clark added a comment -
        • More JavaDoc's to explain things better
        • Scale all cost functions between 0 and 1. This should allow weighting to be more transparent and cleans up some of the math.
        • Added a cost function that takes data locality into consideration.
        • Added a Class to look up data locality. Removing that from the base class. I had to add a cache or the load balance is painfully slow.
        • Cleaned up a lot of code.
        • Added lots more tests for StochasticLoadBalancer.
        Show
        Elliott Clark added a comment - More JavaDoc's to explain things better Scale all cost functions between 0 and 1. This should allow weighting to be more transparent and cleans up some of the math. Added a cost function that takes data locality into consideration. Added a Class to look up data locality. Removing that from the base class. I had to add a cache or the load balance is painfully slow. Cleaned up a lot of code. Added lots more tests for StochasticLoadBalancer.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12526578/HBASE-5959-2.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 13 new or modified tests.

        +1 hadoop23. The patch compiles against the hadoop 0.23.x profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 28 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.io.hfile.TestForceCacheImportantBlocks
        org.apache.hadoop.hbase.replication.TestReplication
        org.apache.hadoop.hbase.TestDrainingServer
        org.apache.hadoop.hbase.master.TestAssignmentManager

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1852//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1852//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1852//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12526578/HBASE-5959-2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 13 new or modified tests. +1 hadoop23. The patch compiles against the hadoop 0.23.x profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 28 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.io.hfile.TestForceCacheImportantBlocks org.apache.hadoop.hbase.replication.TestReplication org.apache.hadoop.hbase.TestDrainingServer org.apache.hadoop.hbase.master.TestAssignmentManager Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1852//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1852//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1852//console This message is automatically generated.
        Hide
        Elliott Clark added a comment -

        Patch to fix the find bugs that were exposed when refactoring.

        The tests all seems to pass locally for me. We'll see if they are just being flakey.

        Show
        Elliott Clark added a comment - Patch to fix the find bugs that were exposed when refactoring. The tests all seems to pass locally for me. We'll see if they are just being flakey.
        Hide
        stack added a comment -

        The move of LoadBalancer down into a package is a no-no given its a Public Interface (see the annotation just above class declaration). Would need to go through a deprecation cycle. Factory can go down into the new balancer package. Thats fine.

        You don't need these lines:

        + * Copyright 2012 The Apache Software Foundation
        

        Is BalanceInfo generically useful or is it useful only to your new balance algo?

        Fix 'It provides the the functions used to by'

        You don't want to pass in master services on construction? You don't want to pass cluster status on invocation of the balance function? Rather than use setters (when setters, no insurance when they will be called)

        Need class comment on new class ClusterLoadState. Does it need to be public? Do methods in this class need to be public? Can they be package protected?

        Did you just move the old classes or do you do some changes to them (I did not review for changes).

        RegionInfoComparator seems like a misleading name given what it does? Indent is not the two spaces used in rest of code base.

        How often will we run this block location calcuation? Its expensive. We need to be careful.

        RegionPlanCompartor should be inner class of RegionPlan?

        Will do a deeper review later but so far, looks great (need to look at your new doohickey balancer more)

        Show
        stack added a comment - The move of LoadBalancer down into a package is a no-no given its a Public Interface (see the annotation just above class declaration). Would need to go through a deprecation cycle. Factory can go down into the new balancer package. Thats fine. You don't need these lines: + * Copyright 2012 The Apache Software Foundation Is BalanceInfo generically useful or is it useful only to your new balance algo? Fix 'It provides the the functions used to by' You don't want to pass in master services on construction? You don't want to pass cluster status on invocation of the balance function? Rather than use setters (when setters, no insurance when they will be called) Need class comment on new class ClusterLoadState. Does it need to be public? Do methods in this class need to be public? Can they be package protected? Did you just move the old classes or do you do some changes to them (I did not review for changes). RegionInfoComparator seems like a misleading name given what it does? Indent is not the two spaces used in rest of code base. How often will we run this block location calcuation? Its expensive. We need to be careful. RegionPlanCompartor should be inner class of RegionPlan? Will do a deeper review later but so far, looks great (need to look at your new doohickey balancer more)
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12526592/HBASE-5959-3.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 13 new or modified tests.

        +1 hadoop23. The patch compiles against the hadoop 0.23.x profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 27 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.TestDrainingServer

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1855//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1855//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1855//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12526592/HBASE-5959-3.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 13 new or modified tests. +1 hadoop23. The patch compiles against the hadoop 0.23.x profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 27 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.TestDrainingServer Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1855//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1855//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1855//console This message is automatically generated.
        Hide
        Ted Yu added a comment -

        For a patch of this size, review board should be used.
        Below is part 1 of my review.

        It is necessary to see the benefit of stochastic balancer in a real cluster compared with existing balancer.

        TestRegionRebalancing is an important test which currently exercises default balancer. It would be nice to apply new balancer there - or create a similar test class.

        + * {@link AssignmentManager} to assign regions in the edge cases. It doesn't
        

        Are edge cases the only ones covered by BaseLoadBalancer ?

        +public class ClusterLoadState {
        

        Please add javadoc for the above class.

        +  private final NavigableMap<ServerAndLoad, List<HRegionInfo>> serversByLoad;
        

        The above map doesn't contain region load information. Can region load be added ?

        +  protected List<ServerName> getInternalTopBlockLocations( HRegionInfo region) {
        

        A better name for the above method would be internalGetTopBlockLocations(). Remove the space after '('.

        +   * @param fs
        +   *          the filesystem
        

        There is no need to put parameter explanation on separate line.

        There're two almost identical getTableDescriptor() methods in master package:

          private HTableDescriptor getTableDescriptor(byte[] tableName)
        src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java
          private HTableDescriptor getTableDescriptor(byte[] tableName)
        src/main/java/org/apache/hadoop/hbase/master/DefaultLoadBalancer.java
        

        It would be nice to put it in a utility class.

        For StochasticLoadBalancer:

        + * randomly try and mutate up to the cluster to Cprime. If F(Cprime) < F(C) then
        

        Remove 'up to' above.

        +  public void setMasterServices(MasterServices masterServices) {
        +    this.services = masterServices;
        +  }
        

        RegionLocationFinder has MasterServices member which should be set in the above method.

        +   * should always approach
        

        Please finish the above sentence for balanceCluster().

        +    // No need to balance one cluster.
        +    if (clusterState.size() <= 1) {
        

        The above check should be placed at the beginning of the method. 'one cluster' -> 'cluster with one server'

        +    // Perform a stocastic walk to see if we can get a good fit.
        

        'stocastic' -> 'stochastic'

        +        HRegionInfo lRegion = pickRandmoRegion(leftRegionList, 0);
        +        HRegionInfo rRegion = pickRandmoRegion(rightRegionList, 0.5);
        

        Please explain why selection of move is not symmetrical.

        +    boolean keep = RANDOM.nextFloat() < (0.01 * (1 - scale(0, maxSteps, stepNum)));
        +    return keep;
        

        A single line should be enough - return directly.

        +   * @return
        +   */
        +  private Map<HRegionInfo, ServerName> createRegionMapping(
        

        Please add javadoc for return value.

        +    double regionCountSkewCost = 100 * computeSkewLoadCost(clusterState);
        +    double tableSkewCost = 50 * computeTableSkewLoadCost(clusterState);
        +    double localityCost = 40 * computeDataLocalityCost(clusterState);
        

        May I ask how the above coefficients were determined ? Should user be able to change them through config parameters ?

        For computeSkewLoadCost():

        +    double max = (mean * (clusterState.size() - 1)) + (Math.abs(mean - numRegions));
        

        numRegions is total number of regions, meaning it is greater than mean. The Math.abs() isn't necessary above.
        May I ask how the above max is derived ?

        Show
        Ted Yu added a comment - For a patch of this size, review board should be used. Below is part 1 of my review. It is necessary to see the benefit of stochastic balancer in a real cluster compared with existing balancer. TestRegionRebalancing is an important test which currently exercises default balancer. It would be nice to apply new balancer there - or create a similar test class. + * {@link AssignmentManager} to assign regions in the edge cases. It doesn't Are edge cases the only ones covered by BaseLoadBalancer ? + public class ClusterLoadState { Please add javadoc for the above class. + private final NavigableMap<ServerAndLoad, List<HRegionInfo>> serversByLoad; The above map doesn't contain region load information. Can region load be added ? + protected List<ServerName> getInternalTopBlockLocations( HRegionInfo region) { A better name for the above method would be internalGetTopBlockLocations(). Remove the space after '('. + * @param fs + * the filesystem There is no need to put parameter explanation on separate line. There're two almost identical getTableDescriptor() methods in master package: private HTableDescriptor getTableDescriptor( byte [] tableName) src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java private HTableDescriptor getTableDescriptor( byte [] tableName) src/main/java/org/apache/hadoop/hbase/master/DefaultLoadBalancer.java It would be nice to put it in a utility class. For StochasticLoadBalancer: + * randomly try and mutate up to the cluster to Cprime. If F(Cprime) < F(C) then Remove 'up to' above. + public void setMasterServices(MasterServices masterServices) { + this .services = masterServices; + } RegionLocationFinder has MasterServices member which should be set in the above method. + * should always approach Please finish the above sentence for balanceCluster(). + // No need to balance one cluster. + if (clusterState.size() <= 1) { The above check should be placed at the beginning of the method. 'one cluster' -> 'cluster with one server' + // Perform a stocastic walk to see if we can get a good fit. 'stocastic' -> 'stochastic' + HRegionInfo lRegion = pickRandmoRegion(leftRegionList, 0); + HRegionInfo rRegion = pickRandmoRegion(rightRegionList, 0.5); Please explain why selection of move is not symmetrical. + boolean keep = RANDOM.nextFloat() < (0.01 * (1 - scale(0, maxSteps, stepNum))); + return keep; A single line should be enough - return directly. + * @ return + */ + private Map<HRegionInfo, ServerName> createRegionMapping( Please add javadoc for return value. + double regionCountSkewCost = 100 * computeSkewLoadCost(clusterState); + double tableSkewCost = 50 * computeTableSkewLoadCost(clusterState); + double localityCost = 40 * computeDataLocalityCost(clusterState); May I ask how the above coefficients were determined ? Should user be able to change them through config parameters ? For computeSkewLoadCost(): + double max = (mean * (clusterState.size() - 1)) + ( Math .abs(mean - numRegions)); numRegions is total number of regions, meaning it is greater than mean. The Math.abs() isn't necessary above. May I ask how the above max is derived ?
        Hide
        Ted Yu added a comment -

        For computeTableSkewLoadCost(), normally we use region count (per table) for region servers to judge skew.
        I feel the skew in table count is not a good indicator.

        +   * @return Cost of imbalance in table.
        

        'table' -> 'tables'

        +    for (Entry<ServerName, List<HRegionInfo>> entry : clusterState.entrySet()) {
        +      Set<String> tablesOnRegionServer = new HashSet<String>();
        

        We can clear the HashSet after the following line in the loop (instead of creating HashSet every iteration):

        +      tablesPerServer.add(new Double(tablesOnRegionServer.size()));
        

        Math.abs() is not needed in the following:

        +      skewCost += Math.abs(numTables - count.doubleValue());
        

        For computeDataLocalityCost():

        +        // If we can't find where the data is getTopBlock returns null.
        +        // so count that as being the best possible.
        +        if (dataOnServers == null) {
        

        I think worst should be assumed if block locations cannot be retrieved.

        I asked the coefficients of various costs to be configurable. This is because users tend to give high priority to locality cost component.

        BalancerTestBase.java is missing license.

        + * Class used to be the base of unit tests on load balancers. It gives helper
        

        'Class used to be the base' -> 'Base class'.
        assertClusterAsBalanced() should be package-private.

        For reconcile(),

        +   * This assumes the RegionPlan HSI instances are the same ones in the map, so
        +   * actually no need to even pass in the map, but I think it's clearer.
        

        Please expand HSI to full class name. The two parameters are both Lists. Where is the map ? I think it is Map<ServerName, ServerAndLoad> map.
        Please describe what this method does.

        Show
        Ted Yu added a comment - For computeTableSkewLoadCost(), normally we use region count (per table) for region servers to judge skew. I feel the skew in table count is not a good indicator. + * @ return Cost of imbalance in table. 'table' -> 'tables' + for (Entry<ServerName, List<HRegionInfo>> entry : clusterState.entrySet()) { + Set< String > tablesOnRegionServer = new HashSet< String >(); We can clear the HashSet after the following line in the loop (instead of creating HashSet every iteration): + tablesPerServer.add( new Double (tablesOnRegionServer.size())); Math.abs() is not needed in the following: + skewCost += Math .abs(numTables - count.doubleValue()); For computeDataLocalityCost(): + // If we can't find where the data is getTopBlock returns null . + // so count that as being the best possible. + if (dataOnServers == null ) { I think worst should be assumed if block locations cannot be retrieved. I asked the coefficients of various costs to be configurable. This is because users tend to give high priority to locality cost component. BalancerTestBase.java is missing license. + * Class used to be the base of unit tests on load balancers. It gives helper 'Class used to be the base' -> 'Base class'. assertClusterAsBalanced() should be package-private. For reconcile(), + * This assumes the RegionPlan HSI instances are the same ones in the map, so + * actually no need to even pass in the map, but I think it's clearer. Please expand HSI to full class name. The two parameters are both Lists. Where is the map ? I think it is Map<ServerName, ServerAndLoad> map. Please describe what this method does.
        Hide
        Elliott Clark added a comment -

        I'll put the next version up on a review board.

        Show
        Elliott Clark added a comment - I'll put the next version up on a review board.
        Hide
        Ted Yu added a comment -
        +    double moveCost = computeMoveCost(initialRegionMapping, clusterState);
        +    double regionCountSkewCost = 100 * computeSkewLoadCost(clusterState);
        +    double tableSkewCost = 50 * computeTableSkewLoadCost(clusterState);
        +    double localityCost = 40 * computeDataLocalityCost(clusterState);
        

        Since the value of move cost varies greatly, the coefficients should be designed such that the three latter costs are brought to the same magnitude of move cost. Meaning, constant coefficients may not be the best choice.

        Show
        Ted Yu added a comment - + double moveCost = computeMoveCost(initialRegionMapping, clusterState); + double regionCountSkewCost = 100 * computeSkewLoadCost(clusterState); + double tableSkewCost = 50 * computeTableSkewLoadCost(clusterState); + double localityCost = 40 * computeDataLocalityCost(clusterState); Since the value of move cost varies greatly, the coefficients should be designed such that the three latter costs are brought to the same magnitude of move cost. Meaning, constant coefficients may not be the best choice.
        Hide
        Elliott Clark added a comment -

        Move cost is just there to stop thrashing. We want all the other to be way above what move cost is. If moving were at the same magnitude that regionSkewCost was then the balancer would not work at all. It would only keep the random mutations.

        Since everything is normalized to between 0 and 1 any multiplier(2 or above) could work, but 100 just makes things a little bit easier to reason about. Basically we'll keep a move if it makes the region skew cost better. And then the balancer will search for how to make that move have the best table skew or the best data locality .

        Show
        Elliott Clark added a comment - Move cost is just there to stop thrashing. We want all the other to be way above what move cost is. If moving were at the same magnitude that regionSkewCost was then the balancer would not work at all. It would only keep the random mutations. Since everything is normalized to between 0 and 1 any multiplier(2 or above) could work, but 100 just makes things a little bit easier to reason about. Basically we'll keep a move if it makes the region skew cost better. And then the balancer will search for how to make that move have the best table skew or the best data locality .
        Hide
        Ted Yu added a comment -

        I didn't take into account the scale() call at the end of computeMoveCost().

        I would think data locality cost should be given higher weight.

        Can computeSkewLoadCost() and computeTableSkewLoadCost() be combined ? In default load balancer, regions are balanced per table.
        The way computeTableSkewLoadCost() currently is implemented, one region from a table on region server can satisfy its goal.

        Show
        Ted Yu added a comment - I didn't take into account the scale() call at the end of computeMoveCost(). I would think data locality cost should be given higher weight. Can computeSkewLoadCost() and computeTableSkewLoadCost() be combined ? In default load balancer, regions are balanced per table. The way computeTableSkewLoadCost() currently is implemented, one region from a table on region server can satisfy its goal.
        Hide
        Phabricator added a comment -

        eclark requested code review of "HBASE-5959 [jira] Add other load balancers".
        Reviewers: JIRA

        HBASE-5959 Add new load balancer

        Now that balancers are pluggable we should give some options.b

        TEST PLAN
        EMPTY

        REVISION DETAIL
        https://reviews.facebook.net/D3189

        AFFECTED FILES
        pom.xml
        src/main/java/org/apache/hadoop/hbase/master/HMaster.java
        src/main/java/org/apache/hadoop/hbase/master/LoadBalancer.java
        src/main/java/org/apache/hadoop/hbase/master/RegionPlan.java
        src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java
        src/main/java/org/apache/hadoop/hbase/master/balancer/ClusterLoadState.java
        src/main/java/org/apache/hadoop/hbase/master/balancer/DefaultLoadBalancer.java
        src/main/java/org/apache/hadoop/hbase/master/DefaultLoadBalancer.java
        src/main/java/org/apache/hadoop/hbase/master/balancer/LoadBalancerFactory.java
        src/main/java/org/apache/hadoop/hbase/master/LoadBalancerFactory.java
        src/main/java/org/apache/hadoop/hbase/master/balancer/RegionInfoComparator.java
        src/main/java/org/apache/hadoop/hbase/master/ServerAndLoad.java
        src/main/java/org/apache/hadoop/hbase/master/balancer/RegionLocationFinder.java
        src/main/java/org/apache/hadoop/hbase/master/balancer/ServerAndLoad.java
        src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java
        src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java
        src/test/java/org/apache/hadoop/hbase/master/TestDefaultLoadBalancer.java
        src/test/java/org/apache/hadoop/hbase/master/balancer/BalancerTestBase.java
        src/test/java/org/apache/hadoop/hbase/master/balancer/TestBaseLoadBalancer.java
        src/test/java/org/apache/hadoop/hbase/master/balancer/TestDefaultLoadBalancer.java
        src/test/java/org/apache/hadoop/hbase/master/balancer/TestStochasticLoadBalancer.java

        MANAGE HERALD DIFFERENTIAL RULES
        https://reviews.facebook.net/herald/view/differential/

        WHY DID I GET THIS EMAIL?
        https://reviews.facebook.net/herald/transcript/7245/

        To: JIRA, eclark

        Show
        Phabricator added a comment - eclark requested code review of " HBASE-5959 [jira] Add other load balancers". Reviewers: JIRA HBASE-5959 Add new load balancer Now that balancers are pluggable we should give some options.b TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D3189 AFFECTED FILES pom.xml src/main/java/org/apache/hadoop/hbase/master/HMaster.java src/main/java/org/apache/hadoop/hbase/master/LoadBalancer.java src/main/java/org/apache/hadoop/hbase/master/RegionPlan.java src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java src/main/java/org/apache/hadoop/hbase/master/balancer/ClusterLoadState.java src/main/java/org/apache/hadoop/hbase/master/balancer/DefaultLoadBalancer.java src/main/java/org/apache/hadoop/hbase/master/DefaultLoadBalancer.java src/main/java/org/apache/hadoop/hbase/master/balancer/LoadBalancerFactory.java src/main/java/org/apache/hadoop/hbase/master/LoadBalancerFactory.java src/main/java/org/apache/hadoop/hbase/master/balancer/RegionInfoComparator.java src/main/java/org/apache/hadoop/hbase/master/ServerAndLoad.java src/main/java/org/apache/hadoop/hbase/master/balancer/RegionLocationFinder.java src/main/java/org/apache/hadoop/hbase/master/balancer/ServerAndLoad.java src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java src/test/java/org/apache/hadoop/hbase/master/TestDefaultLoadBalancer.java src/test/java/org/apache/hadoop/hbase/master/balancer/BalancerTestBase.java src/test/java/org/apache/hadoop/hbase/master/balancer/TestBaseLoadBalancer.java src/test/java/org/apache/hadoop/hbase/master/balancer/TestDefaultLoadBalancer.java src/test/java/org/apache/hadoop/hbase/master/balancer/TestStochasticLoadBalancer.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/7245/ To: JIRA, eclark
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12526860/HBASE-5959.D3189.1.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 13 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1867//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12526860/HBASE-5959.D3189.1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 13 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1867//console This message is automatically generated.
        Hide
        Phabricator added a comment -

        eclark updated the revision "HBASE-5959 [jira] Add other load balancers".
        Reviewers: JIRA

        Changed the copyright headers and updated the spelling of stochastic in the property names

        REVISION DETAIL
        https://reviews.facebook.net/D3189

        AFFECTED FILES
        pom.xml
        src/main/java/org/apache/hadoop/hbase/master/HMaster.java
        src/main/java/org/apache/hadoop/hbase/master/LoadBalancer.java
        src/main/java/org/apache/hadoop/hbase/master/RegionPlan.java
        src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java
        src/main/java/org/apache/hadoop/hbase/master/balancer/ClusterLoadState.java
        src/main/java/org/apache/hadoop/hbase/master/balancer/DefaultLoadBalancer.java
        src/main/java/org/apache/hadoop/hbase/master/DefaultLoadBalancer.java
        src/main/java/org/apache/hadoop/hbase/master/balancer/LoadBalancerFactory.java
        src/main/java/org/apache/hadoop/hbase/master/LoadBalancerFactory.java
        src/main/java/org/apache/hadoop/hbase/master/balancer/RegionInfoComparator.java
        src/main/java/org/apache/hadoop/hbase/master/ServerAndLoad.java
        src/main/java/org/apache/hadoop/hbase/master/balancer/RegionLocationFinder.java
        src/main/java/org/apache/hadoop/hbase/master/balancer/ServerAndLoad.java
        src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java
        src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java
        src/test/java/org/apache/hadoop/hbase/master/TestDefaultLoadBalancer.java
        src/test/java/org/apache/hadoop/hbase/master/balancer/BalancerTestBase.java
        src/test/java/org/apache/hadoop/hbase/master/balancer/TestBaseLoadBalancer.java
        src/test/java/org/apache/hadoop/hbase/master/balancer/TestDefaultLoadBalancer.java
        src/test/java/org/apache/hadoop/hbase/master/balancer/TestStochasticLoadBalancer.java

        To: JIRA, eclark

        Show
        Phabricator added a comment - eclark updated the revision " HBASE-5959 [jira] Add other load balancers". Reviewers: JIRA Changed the copyright headers and updated the spelling of stochastic in the property names REVISION DETAIL https://reviews.facebook.net/D3189 AFFECTED FILES pom.xml src/main/java/org/apache/hadoop/hbase/master/HMaster.java src/main/java/org/apache/hadoop/hbase/master/LoadBalancer.java src/main/java/org/apache/hadoop/hbase/master/RegionPlan.java src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java src/main/java/org/apache/hadoop/hbase/master/balancer/ClusterLoadState.java src/main/java/org/apache/hadoop/hbase/master/balancer/DefaultLoadBalancer.java src/main/java/org/apache/hadoop/hbase/master/DefaultLoadBalancer.java src/main/java/org/apache/hadoop/hbase/master/balancer/LoadBalancerFactory.java src/main/java/org/apache/hadoop/hbase/master/LoadBalancerFactory.java src/main/java/org/apache/hadoop/hbase/master/balancer/RegionInfoComparator.java src/main/java/org/apache/hadoop/hbase/master/ServerAndLoad.java src/main/java/org/apache/hadoop/hbase/master/balancer/RegionLocationFinder.java src/main/java/org/apache/hadoop/hbase/master/balancer/ServerAndLoad.java src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java src/test/java/org/apache/hadoop/hbase/master/TestDefaultLoadBalancer.java src/test/java/org/apache/hadoop/hbase/master/balancer/BalancerTestBase.java src/test/java/org/apache/hadoop/hbase/master/balancer/TestBaseLoadBalancer.java src/test/java/org/apache/hadoop/hbase/master/balancer/TestDefaultLoadBalancer.java src/test/java/org/apache/hadoop/hbase/master/balancer/TestStochasticLoadBalancer.java To: JIRA, eclark
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12526862/HBASE-5959.D3189.2.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 13 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1868//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12526862/HBASE-5959.D3189.2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 13 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1868//console This message is automatically generated.
        Hide
        Phabricator added a comment -

        eclark updated the revision "HBASE-5959 [jira] Add other load balancers".
        Reviewers: JIRA

        Seems like the linear nature of our problem means that the random probability of keeping a solution is not needed.

        I also addeded scaling of the number of iterations needed before accepting a solution. This should help smaller cluster balance faster.

        REVISION DETAIL
        https://reviews.facebook.net/D3189

        AFFECTED FILES
        pom.xml
        src/main/java/org/apache/hadoop/hbase/master/HMaster.java
        src/main/java/org/apache/hadoop/hbase/master/LoadBalancer.java
        src/main/java/org/apache/hadoop/hbase/master/RegionPlan.java
        src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java
        src/main/java/org/apache/hadoop/hbase/master/balancer/ClusterLoadState.java
        src/main/java/org/apache/hadoop/hbase/master/balancer/DefaultLoadBalancer.java
        src/main/java/org/apache/hadoop/hbase/master/DefaultLoadBalancer.java
        src/main/java/org/apache/hadoop/hbase/master/balancer/LoadBalancerFactory.java
        src/main/java/org/apache/hadoop/hbase/master/LoadBalancerFactory.java
        src/main/java/org/apache/hadoop/hbase/master/balancer/RegionInfoComparator.java
        src/main/java/org/apache/hadoop/hbase/master/ServerAndLoad.java
        src/main/java/org/apache/hadoop/hbase/master/balancer/RegionLocationFinder.java
        src/main/java/org/apache/hadoop/hbase/master/balancer/ServerAndLoad.java
        src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java
        src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java
        src/test/java/org/apache/hadoop/hbase/master/TestDefaultLoadBalancer.java
        src/test/java/org/apache/hadoop/hbase/master/balancer/BalancerTestBase.java
        src/test/java/org/apache/hadoop/hbase/master/balancer/TestBaseLoadBalancer.java
        src/test/java/org/apache/hadoop/hbase/master/balancer/TestDefaultLoadBalancer.java
        src/test/java/org/apache/hadoop/hbase/master/balancer/TestStochasticLoadBalancer.java

        To: JIRA, eclark

        Show
        Phabricator added a comment - eclark updated the revision " HBASE-5959 [jira] Add other load balancers". Reviewers: JIRA Seems like the linear nature of our problem means that the random probability of keeping a solution is not needed. I also addeded scaling of the number of iterations needed before accepting a solution. This should help smaller cluster balance faster. REVISION DETAIL https://reviews.facebook.net/D3189 AFFECTED FILES pom.xml src/main/java/org/apache/hadoop/hbase/master/HMaster.java src/main/java/org/apache/hadoop/hbase/master/LoadBalancer.java src/main/java/org/apache/hadoop/hbase/master/RegionPlan.java src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java src/main/java/org/apache/hadoop/hbase/master/balancer/ClusterLoadState.java src/main/java/org/apache/hadoop/hbase/master/balancer/DefaultLoadBalancer.java src/main/java/org/apache/hadoop/hbase/master/DefaultLoadBalancer.java src/main/java/org/apache/hadoop/hbase/master/balancer/LoadBalancerFactory.java src/main/java/org/apache/hadoop/hbase/master/LoadBalancerFactory.java src/main/java/org/apache/hadoop/hbase/master/balancer/RegionInfoComparator.java src/main/java/org/apache/hadoop/hbase/master/ServerAndLoad.java src/main/java/org/apache/hadoop/hbase/master/balancer/RegionLocationFinder.java src/main/java/org/apache/hadoop/hbase/master/balancer/ServerAndLoad.java src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java src/test/java/org/apache/hadoop/hbase/master/TestDefaultLoadBalancer.java src/test/java/org/apache/hadoop/hbase/master/balancer/BalancerTestBase.java src/test/java/org/apache/hadoop/hbase/master/balancer/TestBaseLoadBalancer.java src/test/java/org/apache/hadoop/hbase/master/balancer/TestDefaultLoadBalancer.java src/test/java/org/apache/hadoop/hbase/master/balancer/TestStochasticLoadBalancer.java To: JIRA, eclark
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12526865/HBASE-5959.D3189.3.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 13 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1869//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12526865/HBASE-5959.D3189.3.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 13 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1869//console This message is automatically generated.
        Hide
        Elliott Clark added a comment -

        Not sure why Phabricator is not attaching a good patch. Here is a patch version of what is up on reviews.facebook.net.

        Attaching so that hadoopQA can take a crack at it.

        Show
        Elliott Clark added a comment - Not sure why Phabricator is not attaching a good patch. Here is a patch version of what is up on reviews.facebook.net. Attaching so that hadoopQA can take a crack at it.
        Hide
        Phabricator added a comment -

        eclark updated the revision "HBASE-5959 [jira] Add other load balancers".
        Reviewers: JIRA

        Make TestRegionRebalancing test both DefaultLoadBalancer and the newer stochastic one.

        REVISION DETAIL
        https://reviews.facebook.net/D3189

        AFFECTED FILES
        pom.xml
        src/main/java/org/apache/hadoop/hbase/master/HMaster.java
        src/main/java/org/apache/hadoop/hbase/master/LoadBalancer.java
        src/main/java/org/apache/hadoop/hbase/master/RegionPlan.java
        src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java
        src/main/java/org/apache/hadoop/hbase/master/balancer/ClusterLoadState.java
        src/main/java/org/apache/hadoop/hbase/master/balancer/DefaultLoadBalancer.java
        src/main/java/org/apache/hadoop/hbase/master/DefaultLoadBalancer.java
        src/main/java/org/apache/hadoop/hbase/master/balancer/LoadBalancerFactory.java
        src/main/java/org/apache/hadoop/hbase/master/LoadBalancerFactory.java
        src/main/java/org/apache/hadoop/hbase/master/balancer/RegionInfoComparator.java
        src/main/java/org/apache/hadoop/hbase/master/ServerAndLoad.java
        src/main/java/org/apache/hadoop/hbase/master/balancer/RegionLocationFinder.java
        src/main/java/org/apache/hadoop/hbase/master/balancer/ServerAndLoad.java
        src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java
        src/test/java/org/apache/hadoop/hbase/TestRegionRebalancing.java
        src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java
        src/test/java/org/apache/hadoop/hbase/master/TestDefaultLoadBalancer.java
        src/test/java/org/apache/hadoop/hbase/master/balancer/BalancerTestBase.java
        src/test/java/org/apache/hadoop/hbase/master/balancer/TestBaseLoadBalancer.java
        src/test/java/org/apache/hadoop/hbase/master/balancer/TestDefaultLoadBalancer.java
        src/test/java/org/apache/hadoop/hbase/master/balancer/TestStochasticLoadBalancer.java

        To: JIRA, eclark

        Show
        Phabricator added a comment - eclark updated the revision " HBASE-5959 [jira] Add other load balancers". Reviewers: JIRA Make TestRegionRebalancing test both DefaultLoadBalancer and the newer stochastic one. REVISION DETAIL https://reviews.facebook.net/D3189 AFFECTED FILES pom.xml src/main/java/org/apache/hadoop/hbase/master/HMaster.java src/main/java/org/apache/hadoop/hbase/master/LoadBalancer.java src/main/java/org/apache/hadoop/hbase/master/RegionPlan.java src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java src/main/java/org/apache/hadoop/hbase/master/balancer/ClusterLoadState.java src/main/java/org/apache/hadoop/hbase/master/balancer/DefaultLoadBalancer.java src/main/java/org/apache/hadoop/hbase/master/DefaultLoadBalancer.java src/main/java/org/apache/hadoop/hbase/master/balancer/LoadBalancerFactory.java src/main/java/org/apache/hadoop/hbase/master/LoadBalancerFactory.java src/main/java/org/apache/hadoop/hbase/master/balancer/RegionInfoComparator.java src/main/java/org/apache/hadoop/hbase/master/ServerAndLoad.java src/main/java/org/apache/hadoop/hbase/master/balancer/RegionLocationFinder.java src/main/java/org/apache/hadoop/hbase/master/balancer/ServerAndLoad.java src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java src/test/java/org/apache/hadoop/hbase/TestRegionRebalancing.java src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java src/test/java/org/apache/hadoop/hbase/master/TestDefaultLoadBalancer.java src/test/java/org/apache/hadoop/hbase/master/balancer/BalancerTestBase.java src/test/java/org/apache/hadoop/hbase/master/balancer/TestBaseLoadBalancer.java src/test/java/org/apache/hadoop/hbase/master/balancer/TestDefaultLoadBalancer.java src/test/java/org/apache/hadoop/hbase/master/balancer/TestStochasticLoadBalancer.java To: JIRA, eclark
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12527489/HBASE-5959.D3189.4.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 16 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1873//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12527489/HBASE-5959.D3189.4.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 16 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1873//console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12527489/HBASE-5959.D3189.4.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 16 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1874//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12527489/HBASE-5959.D3189.4.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 16 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1874//console This message is automatically generated.
        Hide
        Elliott Clark added a comment -

        Patch version again.

        Show
        Elliott Clark added a comment - Patch version again.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12527497/HBASE-5959-7.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 16 new or modified tests.

        +1 hadoop23. The patch compiles against the hadoop 0.23.x profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 33 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1876//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1876//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1876//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12527497/HBASE-5959-7.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 16 new or modified tests. +1 hadoop23. The patch compiles against the hadoop 0.23.x profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 33 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1876//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1876//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1876//console This message is automatically generated.
        Hide
        Ted Yu added a comment -

        @Elliot:
        Were all review comments addressed ?

        Feel free to enumerate the ones you have different opinion with.

        Show
        Ted Yu added a comment - @Elliot: Were all review comments addressed ? Feel free to enumerate the ones you have different opinion with.
        Hide
        Elliott Clark added a comment -

        And I didn't move the getTopBlockLocations into a helper as this is already a big enough patch and I wanted to keep the impact to a minimum. If this gets accepted i'll be filing tickets to add region load cost function (region load seems broken right now i'm looking into it), and to refactor getTopBlockLocations.

        Assuming the highest cost or the lowest cost for things has no functional difference. However when/if region load cost functions are added assuming the worst would not work, so I think it's better to start now with always assuming the best cost unless it can be proven otherwise.

        Other than that yes I believe all of the issues were addressed by what is up on phabricator.

        Show
        Elliott Clark added a comment - And I didn't move the getTopBlockLocations into a helper as this is already a big enough patch and I wanted to keep the impact to a minimum. If this gets accepted i'll be filing tickets to add region load cost function (region load seems broken right now i'm looking into it), and to refactor getTopBlockLocations. Assuming the highest cost or the lowest cost for things has no functional difference. However when/if region load cost functions are added assuming the worst would not work, so I think it's better to start now with always assuming the best cost unless it can be proven otherwise. Other than that yes I believe all of the issues were addressed by what is up on phabricator.
        Hide
        Ted Yu added a comment -

        Allow me some time to comment on phabricator.

        Meanwhile, do you have performance numbers for stochastic balancer on a real cluster ? e.g. how long it takes to balance N regions, etc.

        Show
        Ted Yu added a comment - Allow me some time to comment on phabricator. Meanwhile, do you have performance numbers for stochastic balancer on a real cluster ? e.g. how long it takes to balance N regions, etc.
        Hide
        Elliott Clark added a comment -

        hbase(main):004:0> status
        13 servers, 0 dead, 3.6154 average load

        hbase(main):005:0> balancer
        true
        0 row(s) in 3.3460 seconds

        hbase(main):006:0> balancer
        true
        0 row(s) in 3.3530 seconds

        hbase(main):007:0> balancer
        true
        0 row(s) in 3.3160 seconds

        hbase(main):008:0> balancer
        true
        0 row(s) in 3.3850 seconds

        hbase(main):009:0> balancer
        true
        0 row(s) in 3.3580 seconds

        hbase(main):010:0> balancer
        true
        0 row(s) in 3.3190 seconds

        The maxSteps config is nice as it places an upper bounds on the time a large cluster will run the rebalance function. It also allows people to slowly balance their cluster meaning that fewer regions will be moved at a time as the cluster approaches the final solution.

        Show
        Elliott Clark added a comment - hbase(main):004:0> status 13 servers, 0 dead, 3.6154 average load hbase(main):005:0> balancer true 0 row(s) in 3.3460 seconds hbase(main):006:0> balancer true 0 row(s) in 3.3530 seconds hbase(main):007:0> balancer true 0 row(s) in 3.3160 seconds hbase(main):008:0> balancer true 0 row(s) in 3.3850 seconds hbase(main):009:0> balancer true 0 row(s) in 3.3580 seconds hbase(main):010:0> balancer true 0 row(s) in 3.3190 seconds The maxSteps config is nice as it places an upper bounds on the time a large cluster will run the rebalance function. It also allows people to slowly balance their cluster meaning that fewer regions will be moved at a time as the cluster approaches the final solution.
        Hide
        Ted Yu added a comment - - edited

        I was looking for output in master log similar to the following:

            LOG.info("Done. Calculated a load balance in " + (endTime-startTime) + "ms. " +
                "Moving " + totalNumMoved + " regions off of " +
                serversOverloaded + " overloaded servers onto " +
                serversUnderloaded + " less loaded servers. regions to return:" +
                regionsToReturn.size());
        

        w.r.t. maxSteps, is it possible to deduce the maximum time allowable in one round of stochastic balancing ?
        Currently we have (in HMaster.balance()):

            int maximumBalanceTime = getBalancerCutoffTime();
            long cutoffTime = System.currentTimeMillis() + maximumBalanceTime;
        

        I am thinking of the following:

            int maxPlanGenerationTime = maximumBalanceTime *
            // the maximum percentage for planning region rebalance
            getConfiguration().getInt("hbase.balancer.max.planning.percent", 30) / 100;
        ...
                List<RegionPlan> partialPlans = this.balancer.balanceCluster(assignments,
                    maxPlanGenerationTime);
        
        Show
        Ted Yu added a comment - - edited I was looking for output in master log similar to the following: LOG.info( "Done. Calculated a load balance in " + (endTime-startTime) + "ms. " + "Moving " + totalNumMoved + " regions off of " + serversOverloaded + " overloaded servers onto " + serversUnderloaded + " less loaded servers. regions to return :" + regionsToReturn.size()); w.r.t. maxSteps, is it possible to deduce the maximum time allowable in one round of stochastic balancing ? Currently we have (in HMaster.balance()): int maximumBalanceTime = getBalancerCutoffTime(); long cutoffTime = System .currentTimeMillis() + maximumBalanceTime; I am thinking of the following: int maxPlanGenerationTime = maximumBalanceTime * // the maximum percentage for planning region rebalance getConfiguration().getInt( "hbase.balancer.max.planning.percent" , 30) / 100; ... List<RegionPlan> partialPlans = this .balancer.balanceCluster(assignments, maxPlanGenerationTime);
        Hide
        Elliott Clark added a comment -

        The new balancer doesn't have that logging. So there's no timing in the logs. I can add that if there are people that need that info in that format.

        No since we iterate through all servers there is no given amount of time. It could be that an iteration could:

        • hit a cold hfile location cache
        • choose not to move regions.
        • or do a move.

        All of those things will take a different amount of time; so we can't really estimate the amount of time an iteration will take. Hence we would have to go back to the timer every iteration through. That would be very costly.

        Show
        Elliott Clark added a comment - The new balancer doesn't have that logging. So there's no timing in the logs. I can add that if there are people that need that info in that format. No since we iterate through all servers there is no given amount of time. It could be that an iteration could: hit a cold hfile location cache choose not to move regions. or do a move. All of those things will take a different amount of time; so we can't really estimate the amount of time an iteration will take. Hence we would have to go back to the timer every iteration through. That would be very costly.
        Hide
        Ted Yu added a comment -

        need that info in that format.

        Please add such information. Of particular interest is the timing information of the first several balancing rounds when block location cache is cold.

        we can't really estimate the amount of time an iteration will take.

        Agreed.

        Hence we would have to go back to the timer every iteration through. That would be very costly.

        Since the maximum time allowable in one iteration is passed to balancer, why would using local timer to gauge elapsed time be expensive ?

        Show
        Ted Yu added a comment - need that info in that format. Please add such information. Of particular interest is the timing information of the first several balancing rounds when block location cache is cold. we can't really estimate the amount of time an iteration will take. Agreed. Hence we would have to go back to the timer every iteration through. That would be very costly. Since the maximum time allowable in one iteration is passed to balancer, why would using local timer to gauge elapsed time be expensive ?
        Hide
        Elliott Clark added a comment -

        We would have to check sys time every loop through and that would require nanotime to have any real accuracy; nannotime is very expensive.

        There is currently no soft realtime requirement. It doesn't seem like adding that would really be worth the extra complexity until there is a big need or lots of people asking for it. Also Stack didn't want me to change any of the public interfaces in this issue. If it becomes a big thing we can file an issue and add soft realtime.

        Show
        Elliott Clark added a comment - We would have to check sys time every loop through and that would require nanotime to have any real accuracy; nannotime is very expensive. There is currently no soft realtime requirement. It doesn't seem like adding that would really be worth the extra complexity until there is a big need or lots of people asking for it. Also Stack didn't want me to change any of the public interfaces in this issue. If it becomes a big thing we can file an issue and add soft realtime.
        Hide
        Phabricator added a comment -

        tedyu has commented on the revision "HBASE-5959 [jira] Add other load balancers".

        More comments to follow.

        INLINE COMMENTS
        src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:251 Do we need to re-fetch these config parameters in each iteration ?
        src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:44 There're extraneous empty lines such as this one.

        Please remove them.
        src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:78 'one cluster' -> 'cluster with one server'
        src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:80 "it's" -> "cluster has"
        src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:197 'use' -> 'uses'
        src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:202 Please add javadoc
        src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:204 Typo: 'pickRandmoRegion' -> 'pickRandomRegion'
        src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:144 Remove empty line.
        src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:154 'plan' -> 'plans'
        src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:224 Please finish the sentence.
        src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:226 Specify what is returned.
        src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:241 'balancer.stochastic' -> 'stochastic.balancer'
        src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:256 I think locality cost should be given higher weight.

        REVISION DETAIL
        https://reviews.facebook.net/D3189

        To: JIRA, eclark
        Cc: tedyu

        Show
        Phabricator added a comment - tedyu has commented on the revision " HBASE-5959 [jira] Add other load balancers". More comments to follow. INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:251 Do we need to re-fetch these config parameters in each iteration ? src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:44 There're extraneous empty lines such as this one. Please remove them. src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:78 'one cluster' -> 'cluster with one server' src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:80 "it's" -> "cluster has" src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:197 'use' -> 'uses' src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:202 Please add javadoc src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:204 Typo: 'pickRandmoRegion' -> 'pickRandomRegion' src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:144 Remove empty line. src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:154 'plan' -> 'plans' src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:224 Please finish the sentence. src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:226 Specify what is returned. src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:241 'balancer.stochastic' -> 'stochastic.balancer' src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:256 I think locality cost should be given higher weight. REVISION DETAIL https://reviews.facebook.net/D3189 To: JIRA, eclark Cc: tedyu
        Hide
        Phabricator added a comment -

        eclark has commented on the revision "HBASE-5959 [jira] Add other load balancers".

        I'll get the config storing in the next version

        INLINE COMMENTS
        src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:241 There are already config paramaters with balancer.<configName> we should keep the hierarchy.
        src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:256 And I actually think it should be given a lower weight as the table cost is something that represents an on going cost to the cluster and the locality cost is a one time transfer. The impetus for doing this work was that we saw production clusters that were no balancing because the old balancer did not want to move regions around to balance the number of regions per table (Something I've seen on several different production clusters now; a real issue on anything that has big batch jobs). Placing locality's weight above table cost would just mean that the same thing could happen.

        However I think the middle ground combined with the ability to change per install is enough without testing on production clusters.
        src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:251 Nope I'll add that in my next version.

        REVISION DETAIL
        https://reviews.facebook.net/D3189

        To: JIRA, eclark
        Cc: tedyu

        Show
        Phabricator added a comment - eclark has commented on the revision " HBASE-5959 [jira] Add other load balancers". I'll get the config storing in the next version INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:241 There are already config paramaters with balancer.<configName> we should keep the hierarchy. src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:256 And I actually think it should be given a lower weight as the table cost is something that represents an on going cost to the cluster and the locality cost is a one time transfer. The impetus for doing this work was that we saw production clusters that were no balancing because the old balancer did not want to move regions around to balance the number of regions per table (Something I've seen on several different production clusters now; a real issue on anything that has big batch jobs). Placing locality's weight above table cost would just mean that the same thing could happen. However I think the middle ground combined with the ability to change per install is enough without testing on production clusters. src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:251 Nope I'll add that in my next version. REVISION DETAIL https://reviews.facebook.net/D3189 To: JIRA, eclark Cc: tedyu
        Hide
        Ted Yu added a comment -

        we saw production clusters that were no balancing because the old balancer did not want to move regions around to balance the number of regions per table

        I think the above was due to lack of per table load balancing in 0.92

        Show
        Ted Yu added a comment - we saw production clusters that were no balancing because the old balancer did not want to move regions around to balance the number of regions per table I think the above was due to lack of per table load balancing in 0.92
        Hide
        Elliott Clark added a comment -

        Yeah; Though I've been testing this with per table turned off as that's where this loadbalancer does it's best, when it has the whole knowledge of everything.

        Show
        Elliott Clark added a comment - Yeah; Though I've been testing this with per table turned off as that's where this loadbalancer does it's best, when it has the whole knowledge of everything.
        Hide
        Phabricator added a comment -

        eclark updated the revision "HBASE-5959 [jira] Add other load balancers".
        Reviewers: JIRA

        More debug output and some small wording/spelling

        REVISION DETAIL
        https://reviews.facebook.net/D3189

        AFFECTED FILES
        pom.xml
        src/main/java/org/apache/hadoop/hbase/master/HMaster.java
        src/main/java/org/apache/hadoop/hbase/master/LoadBalancer.java
        src/main/java/org/apache/hadoop/hbase/master/RegionPlan.java
        src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java
        src/main/java/org/apache/hadoop/hbase/master/balancer/ClusterLoadState.java
        src/main/java/org/apache/hadoop/hbase/master/balancer/DefaultLoadBalancer.java
        src/main/java/org/apache/hadoop/hbase/master/DefaultLoadBalancer.java
        src/main/java/org/apache/hadoop/hbase/master/balancer/LoadBalancerFactory.java
        src/main/java/org/apache/hadoop/hbase/master/LoadBalancerFactory.java
        src/main/java/org/apache/hadoop/hbase/master/balancer/RegionInfoComparator.java
        src/main/java/org/apache/hadoop/hbase/master/ServerAndLoad.java
        src/main/java/org/apache/hadoop/hbase/master/balancer/RegionLocationFinder.java
        src/main/java/org/apache/hadoop/hbase/master/balancer/ServerAndLoad.java
        src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java
        src/test/java/org/apache/hadoop/hbase/TestRegionRebalancing.java
        src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java
        src/test/java/org/apache/hadoop/hbase/master/TestDefaultLoadBalancer.java
        src/test/java/org/apache/hadoop/hbase/master/balancer/BalancerTestBase.java
        src/test/java/org/apache/hadoop/hbase/master/balancer/TestBaseLoadBalancer.java
        src/test/java/org/apache/hadoop/hbase/master/balancer/TestDefaultLoadBalancer.java
        src/test/java/org/apache/hadoop/hbase/master/balancer/TestStochasticLoadBalancer.java

        To: JIRA, eclark
        Cc: tedyu

        Show
        Phabricator added a comment - eclark updated the revision " HBASE-5959 [jira] Add other load balancers". Reviewers: JIRA More debug output and some small wording/spelling REVISION DETAIL https://reviews.facebook.net/D3189 AFFECTED FILES pom.xml src/main/java/org/apache/hadoop/hbase/master/HMaster.java src/main/java/org/apache/hadoop/hbase/master/LoadBalancer.java src/main/java/org/apache/hadoop/hbase/master/RegionPlan.java src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java src/main/java/org/apache/hadoop/hbase/master/balancer/ClusterLoadState.java src/main/java/org/apache/hadoop/hbase/master/balancer/DefaultLoadBalancer.java src/main/java/org/apache/hadoop/hbase/master/DefaultLoadBalancer.java src/main/java/org/apache/hadoop/hbase/master/balancer/LoadBalancerFactory.java src/main/java/org/apache/hadoop/hbase/master/LoadBalancerFactory.java src/main/java/org/apache/hadoop/hbase/master/balancer/RegionInfoComparator.java src/main/java/org/apache/hadoop/hbase/master/ServerAndLoad.java src/main/java/org/apache/hadoop/hbase/master/balancer/RegionLocationFinder.java src/main/java/org/apache/hadoop/hbase/master/balancer/ServerAndLoad.java src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java src/test/java/org/apache/hadoop/hbase/TestRegionRebalancing.java src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java src/test/java/org/apache/hadoop/hbase/master/TestDefaultLoadBalancer.java src/test/java/org/apache/hadoop/hbase/master/balancer/BalancerTestBase.java src/test/java/org/apache/hadoop/hbase/master/balancer/TestBaseLoadBalancer.java src/test/java/org/apache/hadoop/hbase/master/balancer/TestDefaultLoadBalancer.java src/test/java/org/apache/hadoop/hbase/master/balancer/TestStochasticLoadBalancer.java To: JIRA, eclark Cc: tedyu
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12527726/HBASE-5959.D3189.5.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 16 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1901//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12527726/HBASE-5959.D3189.5.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 16 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1901//console This message is automatically generated.
        Hide
        Phabricator added a comment -

        eclark updated the revision "HBASE-5959 [jira] Add other load balancers".
        Reviewers: JIRA

        Changed some log levels and added a return where it was missing.

        REVISION DETAIL
        https://reviews.facebook.net/D3189

        AFFECTED FILES
        pom.xml
        src/main/java/org/apache/hadoop/hbase/master/HMaster.java
        src/main/java/org/apache/hadoop/hbase/master/LoadBalancer.java
        src/main/java/org/apache/hadoop/hbase/master/RegionPlan.java
        src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java
        src/main/java/org/apache/hadoop/hbase/master/balancer/ClusterLoadState.java
        src/main/java/org/apache/hadoop/hbase/master/balancer/DefaultLoadBalancer.java
        src/main/java/org/apache/hadoop/hbase/master/DefaultLoadBalancer.java
        src/main/java/org/apache/hadoop/hbase/master/balancer/LoadBalancerFactory.java
        src/main/java/org/apache/hadoop/hbase/master/LoadBalancerFactory.java
        src/main/java/org/apache/hadoop/hbase/master/balancer/RegionInfoComparator.java
        src/main/java/org/apache/hadoop/hbase/master/ServerAndLoad.java
        src/main/java/org/apache/hadoop/hbase/master/balancer/RegionLocationFinder.java
        src/main/java/org/apache/hadoop/hbase/master/balancer/ServerAndLoad.java
        src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java
        src/test/java/org/apache/hadoop/hbase/TestRegionRebalancing.java
        src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java
        src/test/java/org/apache/hadoop/hbase/master/TestDefaultLoadBalancer.java
        src/test/java/org/apache/hadoop/hbase/master/balancer/BalancerTestBase.java
        src/test/java/org/apache/hadoop/hbase/master/balancer/TestBaseLoadBalancer.java
        src/test/java/org/apache/hadoop/hbase/master/balancer/TestDefaultLoadBalancer.java
        src/test/java/org/apache/hadoop/hbase/master/balancer/TestStochasticLoadBalancer.java

        To: JIRA, eclark
        Cc: tedyu

        Show
        Phabricator added a comment - eclark updated the revision " HBASE-5959 [jira] Add other load balancers". Reviewers: JIRA Changed some log levels and added a return where it was missing. REVISION DETAIL https://reviews.facebook.net/D3189 AFFECTED FILES pom.xml src/main/java/org/apache/hadoop/hbase/master/HMaster.java src/main/java/org/apache/hadoop/hbase/master/LoadBalancer.java src/main/java/org/apache/hadoop/hbase/master/RegionPlan.java src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java src/main/java/org/apache/hadoop/hbase/master/balancer/ClusterLoadState.java src/main/java/org/apache/hadoop/hbase/master/balancer/DefaultLoadBalancer.java src/main/java/org/apache/hadoop/hbase/master/DefaultLoadBalancer.java src/main/java/org/apache/hadoop/hbase/master/balancer/LoadBalancerFactory.java src/main/java/org/apache/hadoop/hbase/master/LoadBalancerFactory.java src/main/java/org/apache/hadoop/hbase/master/balancer/RegionInfoComparator.java src/main/java/org/apache/hadoop/hbase/master/ServerAndLoad.java src/main/java/org/apache/hadoop/hbase/master/balancer/RegionLocationFinder.java src/main/java/org/apache/hadoop/hbase/master/balancer/ServerAndLoad.java src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java src/test/java/org/apache/hadoop/hbase/TestRegionRebalancing.java src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java src/test/java/org/apache/hadoop/hbase/master/TestDefaultLoadBalancer.java src/test/java/org/apache/hadoop/hbase/master/balancer/BalancerTestBase.java src/test/java/org/apache/hadoop/hbase/master/balancer/TestBaseLoadBalancer.java src/test/java/org/apache/hadoop/hbase/master/balancer/TestDefaultLoadBalancer.java src/test/java/org/apache/hadoop/hbase/master/balancer/TestStochasticLoadBalancer.java To: JIRA, eclark Cc: tedyu
        Hide
        Phabricator added a comment -

        eclark updated the revision "HBASE-5959 [jira] Add other load balancers".
        Reviewers: JIRA

        Rebase to current trunk

        REVISION DETAIL
        https://reviews.facebook.net/D3189

        AFFECTED FILES
        pom.xml
        src/main/java/org/apache/hadoop/hbase/master/HMaster.java
        src/main/java/org/apache/hadoop/hbase/master/LoadBalancer.java
        src/main/java/org/apache/hadoop/hbase/master/RegionPlan.java
        src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java
        src/main/java/org/apache/hadoop/hbase/master/balancer/ClusterLoadState.java
        src/main/java/org/apache/hadoop/hbase/master/balancer/DefaultLoadBalancer.java
        src/main/java/org/apache/hadoop/hbase/master/DefaultLoadBalancer.java
        src/main/java/org/apache/hadoop/hbase/master/balancer/LoadBalancerFactory.java
        src/main/java/org/apache/hadoop/hbase/master/LoadBalancerFactory.java
        src/main/java/org/apache/hadoop/hbase/master/balancer/RegionInfoComparator.java
        src/main/java/org/apache/hadoop/hbase/master/ServerAndLoad.java
        src/main/java/org/apache/hadoop/hbase/master/balancer/RegionLocationFinder.java
        src/main/java/org/apache/hadoop/hbase/master/balancer/ServerAndLoad.java
        src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java
        src/test/java/org/apache/hadoop/hbase/TestRegionRebalancing.java
        src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java
        src/test/java/org/apache/hadoop/hbase/master/TestDefaultLoadBalancer.java
        src/test/java/org/apache/hadoop/hbase/master/balancer/BalancerTestBase.java
        src/test/java/org/apache/hadoop/hbase/master/balancer/TestBaseLoadBalancer.java
        src/test/java/org/apache/hadoop/hbase/master/balancer/TestDefaultLoadBalancer.java
        src/test/java/org/apache/hadoop/hbase/master/balancer/TestStochasticLoadBalancer.java

        To: JIRA, eclark
        Cc: tedyu

        Show
        Phabricator added a comment - eclark updated the revision " HBASE-5959 [jira] Add other load balancers". Reviewers: JIRA Rebase to current trunk REVISION DETAIL https://reviews.facebook.net/D3189 AFFECTED FILES pom.xml src/main/java/org/apache/hadoop/hbase/master/HMaster.java src/main/java/org/apache/hadoop/hbase/master/LoadBalancer.java src/main/java/org/apache/hadoop/hbase/master/RegionPlan.java src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java src/main/java/org/apache/hadoop/hbase/master/balancer/ClusterLoadState.java src/main/java/org/apache/hadoop/hbase/master/balancer/DefaultLoadBalancer.java src/main/java/org/apache/hadoop/hbase/master/DefaultLoadBalancer.java src/main/java/org/apache/hadoop/hbase/master/balancer/LoadBalancerFactory.java src/main/java/org/apache/hadoop/hbase/master/LoadBalancerFactory.java src/main/java/org/apache/hadoop/hbase/master/balancer/RegionInfoComparator.java src/main/java/org/apache/hadoop/hbase/master/ServerAndLoad.java src/main/java/org/apache/hadoop/hbase/master/balancer/RegionLocationFinder.java src/main/java/org/apache/hadoop/hbase/master/balancer/ServerAndLoad.java src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java src/test/java/org/apache/hadoop/hbase/TestRegionRebalancing.java src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java src/test/java/org/apache/hadoop/hbase/master/TestDefaultLoadBalancer.java src/test/java/org/apache/hadoop/hbase/master/balancer/BalancerTestBase.java src/test/java/org/apache/hadoop/hbase/master/balancer/TestBaseLoadBalancer.java src/test/java/org/apache/hadoop/hbase/master/balancer/TestDefaultLoadBalancer.java src/test/java/org/apache/hadoop/hbase/master/balancer/TestStochasticLoadBalancer.java To: JIRA, eclark Cc: tedyu
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12527938/HBASE-5959.D3189.7.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 16 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1920//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12527938/HBASE-5959.D3189.7.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 16 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1920//console This message is automatically generated.
        Hide
        Elliott Clark added a comment -

        Added configurable cache timeouts for region location.

        Show
        Elliott Clark added a comment - Added configurable cache timeouts for region location.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12528782/HBASE-5959-8.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 16 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1968//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12528782/HBASE-5959-8.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 16 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1968//console This message is automatically generated.
        Hide
        Elliott Clark added a comment -

        git rebase.

        Show
        Elliott Clark added a comment - git rebase.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12528790/HBASE-5959-9.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 16 new or modified tests.

        +1 hadoop23. The patch compiles against the hadoop 0.23.x profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 35 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.coprocessor.TestMasterObserver
        org.apache.hadoop.hbase.replication.TestReplication
        org.apache.hadoop.hbase.replication.TestMultiSlaveReplication
        org.apache.hadoop.hbase.replication.TestMasterReplication

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1971//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1971//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1971//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12528790/HBASE-5959-9.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 16 new or modified tests. +1 hadoop23. The patch compiles against the hadoop 0.23.x profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 35 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.coprocessor.TestMasterObserver org.apache.hadoop.hbase.replication.TestReplication org.apache.hadoop.hbase.replication.TestMultiSlaveReplication org.apache.hadoop.hbase.replication.TestMasterReplication Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1971//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1971//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1971//console This message is automatically generated.
        Hide
        Ted Yu added a comment -

        I will go over the latest patch one more time.

        Please allow one or two days.

        Show
        Ted Yu added a comment - I will go over the latest patch one more time. Please allow one or two days.
        Hide
        Ted Yu added a comment -

        The latest addition of region load costs is limited by the fact that region load currently doesn't reflect moving average.

        I feel that the 4 new multipliers (readRequestMultiplier, etc) make tuning load balancing harder because there're so many moving parts.

        Show
        Ted Yu added a comment - The latest addition of region load costs is limited by the fact that region load currently doesn't reflect moving average. I feel that the 4 new multipliers (readRequestMultiplier, etc) make tuning load balancing harder because there're so many moving parts.
        Hide
        Elliott Clark added a comment -

        Right now the two most variable(read and write) are not used they have a 0 co-efficient. I'll need to add in a lot more plumbing to get a moving average going and this patch is getting a little too long in the tooth as it is.

        They are the whole point (write requests were the original impetus for this patch). Every time someone talks about load balancing they have a different need. That need is not encompassed by one metric or one single best fit. If it was then the default loadbalnacer would be enough.

        Tuning this is a pretty difficult challenge (if the defaults are deviated from), however with the complexity comes a more powerful ability to move read/write/memory load.

        Show
        Elliott Clark added a comment - Right now the two most variable(read and write) are not used they have a 0 co-efficient. I'll need to add in a lot more plumbing to get a moving average going and this patch is getting a little too long in the tooth as it is. They are the whole point (write requests were the original impetus for this patch). Every time someone talks about load balancing they have a different need. That need is not encompassed by one metric or one single best fit. If it was then the default loadbalnacer would be enough. Tuning this is a pretty difficult challenge (if the defaults are deviated from), however with the complexity comes a more powerful ability to move read/write/memory load.
        Hide
        Elliott Clark added a comment -

        copy of the latest version on the review.

        Show
        Elliott Clark added a comment - copy of the latest version on the review.
        Hide
        stack added a comment -

        Why does RegionPlanComparator implemenent Serializable?

        Maybe doc why you are doing this compare on regionid... are you dependent on it being timestamp?

        Why not do 'return r.getRegionInfo().getRegionId() - l.getRegionInfo().getRegionId();' rather than check < 0 and return -1, etc?

        Remove a 'the' from "...It provides the the functions used to by..."

        Should RegionPlanComparator be public? Could it be package private? It needs to be public so the balancer package has access?

        Shouldn't setMasterServices instead be passed to the constructor? Otherwise, could have an instance of a balancer that was not viable, not until after setMasterServices was called (ditto w/ setClusterStatus)?

        So the method roundRobinAssignment is in the base load balancer. Its expected by all load balancer implementations? Its behavior will be same for all?

        Fix 'randmo server.'

        Does ClusterLoadState need to be public? Does it need audience annotation?

        Why a RegionInfoComparator class and then also a comparator in RegionInfo class? Is the latter a mistaken inclusion?

        Comment seems wrong '+ // The cache for where load balancers are located.'

        This is an expensive call? computeHDFSBlocksDistribution? Goes to NN? We do it for every region in cluster every time we balance? We doing to DOS the NN?

        The call to getTableDescriptor should be getting a descriptor from cache of table descriptors maintained by the master, right?

        Why ServerAndLoad implement Serializable? You planning on java serializing it?

        Does StochasticLoadBalancer have to be a public class?

        This stuff looks great. Nice tests.

        Show
        stack added a comment - Why does RegionPlanComparator implemenent Serializable? Maybe doc why you are doing this compare on regionid... are you dependent on it being timestamp? Why not do 'return r.getRegionInfo().getRegionId() - l.getRegionInfo().getRegionId();' rather than check < 0 and return -1, etc? Remove a 'the' from "...It provides the the functions used to by..." Should RegionPlanComparator be public? Could it be package private? It needs to be public so the balancer package has access? Shouldn't setMasterServices instead be passed to the constructor? Otherwise, could have an instance of a balancer that was not viable, not until after setMasterServices was called (ditto w/ setClusterStatus)? So the method roundRobinAssignment is in the base load balancer. Its expected by all load balancer implementations? Its behavior will be same for all? Fix 'randmo server.' Does ClusterLoadState need to be public? Does it need audience annotation? Why a RegionInfoComparator class and then also a comparator in RegionInfo class? Is the latter a mistaken inclusion? Comment seems wrong '+ // The cache for where load balancers are located.' This is an expensive call? computeHDFSBlocksDistribution? Goes to NN? We do it for every region in cluster every time we balance? We doing to DOS the NN? The call to getTableDescriptor should be getting a descriptor from cache of table descriptors maintained by the master, right? Why ServerAndLoad implement Serializable? You planning on java serializing it? Does StochasticLoadBalancer have to be a public class? This stuff looks great. Nice tests.
        Hide
        Elliott Clark added a comment -

        Why does RegionPlanComparator implemenent Serializable?

        Findbugs wants it so that any collection using this can also be serializable. Probably not needed but trying to correct findbugs where I can.

        Maybe doc why you are doing this compare on regionid... are you dependent on it being timestamp?

        Why not do 'return r.getRegionInfo().getRegionId() - l.getRegionInfo().getRegionId();' rather than check < 0 and return -1, etc?

        These are both just moved code from the old balancer. My bet would be the comparator was written that way so the return works without a type cast and if the diff is greater than an int can hold. But not sure.

        Should RegionPlanComparator be public? Could it be package private? It needs to be public so the balancer package has access?

        Yep it needs to be used by the balancer package.

        Comment seems wrong '+ // The cache for where load balancers are located.'

        This is an expensive call? computeHDFSBlocksDistribution? Goes to NN? We do it for every region in cluster every time we balance? We doing to DOS the NN?

        We do this for every region. However we cache this for a given amount of time. Should we up that beyond 30 mins ?

        Why ServerAndLoad implement Serializable? You planning on java serializing it?

        Another findbugs special

        Does StochasticLoadBalancer have to be a public class?

        It makes the factory much easier. Not 100% required but makes it use less reflection to create an instance.

        Show
        Elliott Clark added a comment - Why does RegionPlanComparator implemenent Serializable? Findbugs wants it so that any collection using this can also be serializable. Probably not needed but trying to correct findbugs where I can. Maybe doc why you are doing this compare on regionid... are you dependent on it being timestamp? Why not do 'return r.getRegionInfo().getRegionId() - l.getRegionInfo().getRegionId();' rather than check < 0 and return -1, etc? These are both just moved code from the old balancer. My bet would be the comparator was written that way so the return works without a type cast and if the diff is greater than an int can hold. But not sure. Should RegionPlanComparator be public? Could it be package private? It needs to be public so the balancer package has access? Yep it needs to be used by the balancer package. Comment seems wrong '+ // The cache for where load balancers are located.' This is an expensive call? computeHDFSBlocksDistribution? Goes to NN? We do it for every region in cluster every time we balance? We doing to DOS the NN? We do this for every region. However we cache this for a given amount of time. Should we up that beyond 30 mins ? Why ServerAndLoad implement Serializable? You planning on java serializing it? Another findbugs special Does StochasticLoadBalancer have to be a public class? It makes the factory much easier. Not 100% required but makes it use less reflection to create an instance.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12530097/HBASE-5959-11.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 16 new or modified tests.

        +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2035//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2035//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12530097/HBASE-5959-11.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 16 new or modified tests. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2035//testReport/ Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2035//console This message is automatically generated.
        Hide
        Elliott Clark added a comment -

        Shouldn't setMasterServices instead be passed to the constructor? Otherwise, could have an instance of a balancer that was not viable, not until after setMasterServices was called (ditto w/ setClusterStatus)?

        It would be nice but using the hadoop ReflectionUtils a conf is the only thing that can be passed in. The balancer should work without either of those, though they are always set on finish master initialization.

        Show
        Elliott Clark added a comment - Shouldn't setMasterServices instead be passed to the constructor? Otherwise, could have an instance of a balancer that was not viable, not until after setMasterServices was called (ditto w/ setClusterStatus)? It would be nice but using the hadoop ReflectionUtils a conf is the only thing that can be passed in. The balancer should work without either of those, though they are always set on finish master initialization.
        Hide
        Elliott Clark added a comment -

        Changed around the comments that Stack asked about.

        Show
        Elliott Clark added a comment - Changed around the comments that Stack asked about.
        Hide
        stack added a comment -

        I'm +1 on this patch. Ted?

        Show
        stack added a comment - I'm +1 on this patch. Ted?
        Hide
        Ted Yu added a comment -

        I was trying to see the difference between Diff 10 and Diff 11.
        Due to rebasing, the diff is not easy to read.

        Show
        Ted Yu added a comment - I was trying to see the difference between Diff 10 and Diff 11. Due to rebasing, the diff is not easy to read.
        Hide
        Elliott Clark added a comment -

        Added more top of class comments to explain all of the config options.

        Addressed a few code comments.

        Phabrictor seems to have been broken on the move to modules. I'll file a bug, but in the mean time i'll attach patches here.

        This should be the final patch.

        Show
        Elliott Clark added a comment - Added more top of class comments to explain all of the config options. Addressed a few code comments. Phabrictor seems to have been broken on the move to modules. I'll file a bug, but in the mean time i'll attach patches here. This should be the final patch.
        Hide
        stack added a comment -

        I'll commit this this afternoon unless objection (old behavior is the default).

        Show
        stack added a comment - I'll commit this this afternoon unless objection (old behavior is the default).
        Hide
        Ted Yu added a comment -

        From https://builds.apache.org/job/PreCommit-HBASE-Build/2060/console :

        /home/jenkins/tools/maven/latest/bin/mvn clean test -DskipTests -DHBasePatchProcess > /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/patchprocess/trunkJavacWarnings.txt 2>&1
        Trunk compilation is broken?
        
        Show
        Ted Yu added a comment - From https://builds.apache.org/job/PreCommit-HBASE-Build/2060/console : /home/jenkins/tools/maven/latest/bin/mvn clean test -DskipTests -DHBasePatchProcess > /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/patchprocess/trunkJavacWarnings.txt 2>&1 Trunk compilation is broken?
        Hide
        Elliott Clark added a comment -

        More test asserts.

        Looks like jenkins is having issues. Running tests locally right now.

        Show
        Elliott Clark added a comment - More test asserts. Looks like jenkins is having issues. Running tests locally right now.
        Hide
        Ted Yu added a comment -

        I got the following when running against patch v14:

        testContendedLogRolling(org.apache.hadoop.hbase.regionserver.wal.TestLogRollingNoCluster)  Time elapsed: 0.201 sec  <<< FAILURE!
        java.lang.AssertionError
          at org.junit.Assert.fail(Assert.java:92)
          at org.junit.Assert.assertTrue(Assert.java:43)
          at org.junit.Assert.assertFalse(Assert.java:68)
          at org.junit.Assert.assertFalse(Assert.java:79)
          at org.apache.hadoop.hbase.regionserver.wal.TestLogRollingNoCluster.testContendedLogRolling(TestLogRollingNoCluster.java:78)
        
        Show
        Ted Yu added a comment - I got the following when running against patch v14: testContendedLogRolling(org.apache.hadoop.hbase.regionserver.wal.TestLogRollingNoCluster) Time elapsed: 0.201 sec <<< FAILURE! java.lang.AssertionError at org.junit.Assert.fail(Assert.java:92) at org.junit.Assert.assertTrue(Assert.java:43) at org.junit.Assert.assertFalse(Assert.java:68) at org.junit.Assert.assertFalse(Assert.java:79) at org.apache.hadoop.hbase.regionserver.wal.TestLogRollingNoCluster.testContendedLogRolling(TestLogRollingNoCluster.java:78)
        Hide
        stack added a comment -

        @Ted If you run w/o this patch, that test passes and then if you run w/ this patch it always fails? That test seems totally unrelated to whats going on in this patch.

        Show
        stack added a comment - @Ted If you run w/o this patch, that test passes and then if you run w/ this patch it always fails? That test seems totally unrelated to whats going on in this patch.
        Hide
        Elliott Clark added a comment -

        Just tried it and got a passing test:

        mvn clean compile test -PrunAllTests -DfailIfNoTests=false -Dtest=TestLogRollingNoCluster                                   ruby-1.9.3-p194  [33s] 
        [INFO] Scanning for projects...
        [INFO] ------------------------------------------------------------------------
        [INFO] Reactor Build Order:
        [INFO] 
        [INFO] HBase
        [INFO] HBase - Server
        [INFO] HBase - Assembly
        [INFO] HBase - Site
        [INFO]                                                                         
        [INFO] ------------------------------------------------------------------------
        [INFO] Building HBase 0.95-SNAPSHOT
        [INFO] ------------------------------------------------------------------------
        [INFO] 
        [INFO] --- maven-clean-plugin:2.4.1:clean (default-clean) @ hbase ---
        [INFO] Deleting /Users/eclark/Code/hbase/target
        [INFO] 
        [INFO] --- maven-remote-resources-plugin:1.1:process (default) @ hbase ---
        [INFO] Setting property: classpath.resource.loader.class => 'org.codehaus.plexus.velocity.ContextClassLoaderResourceLoader'.
        [INFO] Setting property: velocimacro.messages.on => 'false'.
        [INFO] Setting property: resource.loader => 'classpath'.
        [INFO] Setting property: resource.manager.logwhenfound => 'false'.
        [INFO] 
        [INFO] --- maven-remote-resources-plugin:1.1:process (default) @ hbase ---
        [INFO]                                                                         
        [INFO] ------------------------------------------------------------------------
        [INFO] Building HBase - Server 0.95-SNAPSHOT
        [INFO] ------------------------------------------------------------------------
        [INFO] 
        [INFO] --- maven-clean-plugin:2.4.1:clean (default-clean) @ hbase-server ---
        [INFO] Deleting /Users/eclark/Code/hbase/hbase-server/target
        [INFO] 
        [INFO] --- avro-maven-plugin:1.5.3:schema (generate-avro-sources) @ hbase-server ---
        [INFO] 
        [INFO] --- avro-maven-plugin:1.5.3:protocol (generate-avro-sources) @ hbase-server ---
        [INFO] 
        [INFO] --- maven-antrun-plugin:1.6:run (generate) @ hbase-server ---
        [INFO] Executing tasks
        
        main:
            [mkdir] Created dir: /Users/eclark/Code/hbase/hbase-server/target/hbase-webapps
             [copy] Copying 4 files to /Users/eclark/Code/hbase/hbase-server/target/hbase-webapps
        12/05/30 13:36:56 INFO mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
            [mkdir] Created dir: /Users/eclark/Code/hbase/hbase-server/target/hbase-webapps/master/WEB-INF
            [mkdir] Created dir: /Users/eclark/Code/hbase/hbase-server/target/hbase-webapps/regionserver/WEB-INF
        [INFO] Executed tasks
        [INFO] 
        [INFO] --- build-helper-maven-plugin:1.5:add-source (jspcSource-packageInfo-Avro-source) @ hbase-server ---
        [INFO] Source directory: /Users/eclark/Code/hbase/hbase-server/target/generated-jamon added.
        [INFO] Source directory: /Users/eclark/Code/hbase/hbase-server/target/generated-sources/java added.
        [INFO] 
        [INFO] --- jamon-maven-plugin:2.3.4:translate (default) @ hbase-server ---
        [INFO] Translating 4 templates from /Users/eclark/Code/hbase/hbase-server/src/main/jamon to /Users/eclark/Code/hbase/hbase-server/target/generated-jamon
        [INFO] 
        [INFO] --- maven-remote-resources-plugin:1.1:process (default) @ hbase-server ---
        [INFO] 
        [INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ hbase-server ---
        [debug] execute contextualize
        [INFO] Using 'UTF-8' encoding to copy filtered resources.
        [INFO] Copying 6 resources
        [INFO] Copying 1 resource
        [INFO] Copying 3 resources
        [INFO] 
        [INFO] --- maven-compiler-plugin:2.0.2:compile (default-compile) @ hbase-server ---
        [INFO] Compiling 766 source files to /Users/eclark/Code/hbase/hbase-server/target/classes
        [INFO] 
        [INFO] --- avro-maven-plugin:1.5.3:schema (generate-avro-sources) @ hbase-server ---
        [INFO] 
        [INFO] --- avro-maven-plugin:1.5.3:protocol (generate-avro-sources) @ hbase-server ---
        [INFO] 
        [INFO] --- maven-antrun-plugin:1.6:run (generate) @ hbase-server ---
        [INFO] Executing tasks
        
        main:
        12/05/30 13:37:19 INFO mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
        [INFO] Executed tasks
        [INFO] 
        [INFO] --- build-helper-maven-plugin:1.5:add-source (jspcSource-packageInfo-Avro-source) @ hbase-server ---
        [INFO] Source directory: /Users/eclark/Code/hbase/hbase-server/target/generated-jamon added.
        [INFO] Source directory: /Users/eclark/Code/hbase/hbase-server/target/generated-sources/java added.
        [INFO] 
        [INFO] --- jamon-maven-plugin:2.3.4:translate (default) @ hbase-server ---
        [INFO] 
        [INFO] --- maven-remote-resources-plugin:1.1:process (default) @ hbase-server ---
        [INFO] 
        [INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ hbase-server ---
        [debug] execute contextualize
        [INFO] Using 'UTF-8' encoding to copy filtered resources.
        [INFO] Copying 6 resources
        [INFO] Copying 1 resource
        [INFO] Copying 3 resources
        [INFO] Copying 3 resources
        [INFO] 
        [INFO] --- maven-compiler-plugin:2.0.2:compile (default-compile) @ hbase-server ---
        [INFO] Compiling 8 source files to /Users/eclark/Code/hbase/hbase-server/target/classes
        [INFO] 
        [INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ hbase-server ---
        [debug] execute contextualize
        [INFO] Using 'UTF-8' encoding to copy filtered resources.
        [INFO] Copying 5 resources
        [INFO] Copying 3 resources
        [INFO] Copying 3 resources
        [INFO] 
        [INFO] --- maven-compiler-plugin:2.0.2:testCompile (default-testCompile) @ hbase-server ---
        [INFO] Compiling 375 source files to /Users/eclark/Code/hbase/hbase-server/target/test-classes
        [INFO] 
        [INFO] --- maven-surefire-plugin:2.12-TRUNK-HBASE-2:test (default-test) @ hbase-server ---
        [INFO] Surefire report directory: /Users/eclark/Code/hbase/hbase-server/target/surefire-reports
        [INFO] Using configured provider org.apache.maven.surefire.junitcore.JUnitCoreProvider
        
        -------------------------------------------------------
         T E S T S
        -------------------------------------------------------
        2012-05-30 13:37:23.778 java[10644:d703] Unable to load realm info from SCDynamicStore
        Running org.apache.hadoop.hbase.regionserver.wal.TestLogRollingNoCluster
        Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.384 sec
        
        Results :
        
        Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
        
        [INFO] 
        [INFO] --- maven-surefire-plugin:2.12-TRUNK-HBASE-2:test (secondPartTestsExecution) @ hbase-server ---
        [INFO] Surefire report directory: /Users/eclark/Code/hbase/hbase-server/target/surefire-reports
        [INFO] Using configured provider org.apache.maven.surefire.junitcore.JUnitCoreProvider
        
        -------------------------------------------------------
         T E S T S
        -------------------------------------------------------
        
        -------------------------------------------------------
         T E S T S
        -------------------------------------------------------
        
        Results :
        
        Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
        
        [INFO]                                                                         
        [INFO] ------------------------------------------------------------------------
        [INFO] Building HBase - Assembly 0.95-SNAPSHOT
        [INFO] ------------------------------------------------------------------------
        [INFO] 
        [INFO] --- maven-clean-plugin:2.4.1:clean (default-clean) @ hbase-assembly ---
        [INFO] Deleting /Users/eclark/Code/hbase/hbase-assembly/target
        [INFO] 
        [INFO] --- maven-antrun-plugin:1.6:run (arc-setup) @ hbase-assembly ---
        [WARNING] Parameter tasks is deprecated, use target instead
        [INFO] Executing tasks
        
        main:
        [INFO] Executed tasks
        [INFO] 
        [INFO] --- maven-remote-resources-plugin:1.1:process (default) @ hbase-assembly ---
        [INFO] 
        [INFO] --- maven-antrun-plugin:1.6:run (arc-setup) @ hbase-assembly ---
        [WARNING] Parameter tasks is deprecated, use target instead
        [INFO] Executing tasks
        
        main:
        [INFO] Executed tasks
        [INFO] 
        [INFO] --- maven-remote-resources-plugin:1.1:process (default) @ hbase-assembly ---
        [INFO]                                                                         
        [INFO] ------------------------------------------------------------------------
        [INFO] Building HBase - Site 0.95-SNAPSHOT
        [INFO] ------------------------------------------------------------------------
        [INFO] 
        [INFO] --- maven-clean-plugin:2.4.1:clean (default-clean) @ hbase-site ---
        [INFO] Deleting /Users/eclark/Code/hbase/hbase-site/target
        [INFO] 
        [INFO] --- maven-remote-resources-plugin:1.1:process (default) @ hbase-site ---
        [INFO] 
        [INFO] --- maven-remote-resources-plugin:1.1:process (default) @ hbase-site ---
        [INFO] ------------------------------------------------------------------------
        [INFO] Reactor Summary:
        [INFO] 
        [INFO] HBase ............................................. SUCCESS [0.703s]
        [INFO] HBase - Server .................................... SUCCESS [33.019s]
        [INFO] HBase - Assembly .................................. SUCCESS [0.621s]
        [INFO] HBase - Site ...................................... SUCCESS [0.860s]
        [INFO] ------------------------------------------------------------------------
        [INFO] BUILD SUCCESS
        [INFO] ------------------------------------------------------------------------
        [INFO] Total time: 35.527s
        [INFO] Finished at: Wed May 30 13:37:28 PDT 2012
        [INFO] Final Memory: 29M/123M
        [INFO] ------------------------------------------------------------------------
        
        Show
        Elliott Clark added a comment - Just tried it and got a passing test: mvn clean compile test -PrunAllTests -DfailIfNoTests= false -Dtest=TestLogRollingNoCluster ruby-1.9.3-p194 [33s] [INFO] Scanning for projects... [INFO] ------------------------------------------------------------------------ [INFO] Reactor Build Order: [INFO] [INFO] HBase [INFO] HBase - Server [INFO] HBase - Assembly [INFO] HBase - Site [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building HBase 0.95-SNAPSHOT [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- maven-clean-plugin:2.4.1:clean ( default -clean) @ hbase --- [INFO] Deleting /Users/eclark/Code/hbase/target [INFO] [INFO] --- maven-remote-resources-plugin:1.1:process ( default ) @ hbase --- [INFO] Setting property: classpath.resource.loader.class => 'org.codehaus.plexus.velocity.ContextClassLoaderResourceLoader'. [INFO] Setting property: velocimacro.messages.on => ' false '. [INFO] Setting property: resource.loader => 'classpath'. [INFO] Setting property: resource.manager.logwhenfound => ' false '. [INFO] [INFO] --- maven-remote-resources-plugin:1.1:process ( default ) @ hbase --- [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building HBase - Server 0.95-SNAPSHOT [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- maven-clean-plugin:2.4.1:clean ( default -clean) @ hbase-server --- [INFO] Deleting /Users/eclark/Code/hbase/hbase-server/target [INFO] [INFO] --- avro-maven-plugin:1.5.3:schema (generate-avro-sources) @ hbase-server --- [INFO] [INFO] --- avro-maven-plugin:1.5.3:protocol (generate-avro-sources) @ hbase-server --- [INFO] [INFO] --- maven-antrun-plugin:1.6:run (generate) @ hbase-server --- [INFO] Executing tasks main: [mkdir] Created dir: /Users/eclark/Code/hbase/hbase-server/target/hbase-webapps [copy] Copying 4 files to /Users/eclark/Code/hbase/hbase-server/target/hbase-webapps 12/05/30 13:36:56 INFO mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog [mkdir] Created dir: /Users/eclark/Code/hbase/hbase-server/target/hbase-webapps/master/WEB-INF [mkdir] Created dir: /Users/eclark/Code/hbase/hbase-server/target/hbase-webapps/regionserver/WEB-INF [INFO] Executed tasks [INFO] [INFO] --- build-helper-maven-plugin:1.5:add-source (jspcSource-packageInfo-Avro-source) @ hbase-server --- [INFO] Source directory: /Users/eclark/Code/hbase/hbase-server/target/generated-jamon added. [INFO] Source directory: /Users/eclark/Code/hbase/hbase-server/target/generated-sources/java added. [INFO] [INFO] --- jamon-maven-plugin:2.3.4:translate ( default ) @ hbase-server --- [INFO] Translating 4 templates from /Users/eclark/Code/hbase/hbase-server/src/main/jamon to /Users/eclark/Code/hbase/hbase-server/target/generated-jamon [INFO] [INFO] --- maven-remote-resources-plugin:1.1:process ( default ) @ hbase-server --- [INFO] [INFO] --- maven-resources-plugin:2.5:resources ( default -resources) @ hbase-server --- [debug] execute contextualize [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 6 resources [INFO] Copying 1 resource [INFO] Copying 3 resources [INFO] [INFO] --- maven-compiler-plugin:2.0.2:compile ( default -compile) @ hbase-server --- [INFO] Compiling 766 source files to /Users/eclark/Code/hbase/hbase-server/target/classes [INFO] [INFO] --- avro-maven-plugin:1.5.3:schema (generate-avro-sources) @ hbase-server --- [INFO] [INFO] --- avro-maven-plugin:1.5.3:protocol (generate-avro-sources) @ hbase-server --- [INFO] [INFO] --- maven-antrun-plugin:1.6:run (generate) @ hbase-server --- [INFO] Executing tasks main: 12/05/30 13:37:19 INFO mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog [INFO] Executed tasks [INFO] [INFO] --- build-helper-maven-plugin:1.5:add-source (jspcSource-packageInfo-Avro-source) @ hbase-server --- [INFO] Source directory: /Users/eclark/Code/hbase/hbase-server/target/generated-jamon added. [INFO] Source directory: /Users/eclark/Code/hbase/hbase-server/target/generated-sources/java added. [INFO] [INFO] --- jamon-maven-plugin:2.3.4:translate ( default ) @ hbase-server --- [INFO] [INFO] --- maven-remote-resources-plugin:1.1:process ( default ) @ hbase-server --- [INFO] [INFO] --- maven-resources-plugin:2.5:resources ( default -resources) @ hbase-server --- [debug] execute contextualize [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 6 resources [INFO] Copying 1 resource [INFO] Copying 3 resources [INFO] Copying 3 resources [INFO] [INFO] --- maven-compiler-plugin:2.0.2:compile ( default -compile) @ hbase-server --- [INFO] Compiling 8 source files to /Users/eclark/Code/hbase/hbase-server/target/classes [INFO] [INFO] --- maven-resources-plugin:2.5:testResources ( default -testResources) @ hbase-server --- [debug] execute contextualize [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 5 resources [INFO] Copying 3 resources [INFO] Copying 3 resources [INFO] [INFO] --- maven-compiler-plugin:2.0.2:testCompile ( default -testCompile) @ hbase-server --- [INFO] Compiling 375 source files to /Users/eclark/Code/hbase/hbase-server/target/test-classes [INFO] [INFO] --- maven-surefire-plugin:2.12-TRUNK-HBASE-2:test ( default -test) @ hbase-server --- [INFO] Surefire report directory: /Users/eclark/Code/hbase/hbase-server/target/surefire-reports [INFO] Using configured provider org.apache.maven.surefire.junitcore.JUnitCoreProvider ------------------------------------------------------- T E S T S ------------------------------------------------------- 2012-05-30 13:37:23.778 java[10644:d703] Unable to load realm info from SCDynamicStore Running org.apache.hadoop.hbase.regionserver.wal.TestLogRollingNoCluster Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.384 sec Results : Tests run: 1, Failures: 0, Errors: 0, Skipped: 0 [INFO] [INFO] --- maven-surefire-plugin:2.12-TRUNK-HBASE-2:test (secondPartTestsExecution) @ hbase-server --- [INFO] Surefire report directory: /Users/eclark/Code/hbase/hbase-server/target/surefire-reports [INFO] Using configured provider org.apache.maven.surefire.junitcore.JUnitCoreProvider ------------------------------------------------------- T E S T S ------------------------------------------------------- ------------------------------------------------------- T E S T S ------------------------------------------------------- Results : Tests run: 0, Failures: 0, Errors: 0, Skipped: 0 [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building HBase - Assembly 0.95-SNAPSHOT [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- maven-clean-plugin:2.4.1:clean ( default -clean) @ hbase-assembly --- [INFO] Deleting /Users/eclark/Code/hbase/hbase-assembly/target [INFO] [INFO] --- maven-antrun-plugin:1.6:run (arc-setup) @ hbase-assembly --- [WARNING] Parameter tasks is deprecated, use target instead [INFO] Executing tasks main: [INFO] Executed tasks [INFO] [INFO] --- maven-remote-resources-plugin:1.1:process ( default ) @ hbase-assembly --- [INFO] [INFO] --- maven-antrun-plugin:1.6:run (arc-setup) @ hbase-assembly --- [WARNING] Parameter tasks is deprecated, use target instead [INFO] Executing tasks main: [INFO] Executed tasks [INFO] [INFO] --- maven-remote-resources-plugin:1.1:process ( default ) @ hbase-assembly --- [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building HBase - Site 0.95-SNAPSHOT [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- maven-clean-plugin:2.4.1:clean ( default -clean) @ hbase-site --- [INFO] Deleting /Users/eclark/Code/hbase/hbase-site/target [INFO] [INFO] --- maven-remote-resources-plugin:1.1:process ( default ) @ hbase-site --- [INFO] [INFO] --- maven-remote-resources-plugin:1.1:process ( default ) @ hbase-site --- [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] HBase ............................................. SUCCESS [0.703s] [INFO] HBase - Server .................................... SUCCESS [33.019s] [INFO] HBase - Assembly .................................. SUCCESS [0.621s] [INFO] HBase - Site ...................................... SUCCESS [0.860s] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 35.527s [INFO] Finished at: Wed May 30 13:37:28 PDT 2012 [INFO] Final Memory: 29M/123M [INFO] ------------------------------------------------------------------------
        Hide
        Ted Yu added a comment -

        The test failure was due to HBASE-5936.
        I updated workspace and it passed.

        Let's see what Hadoop QA tells us:
        https://builds.apache.org/job/PreCommit-HBASE-Build/2062/console

        Show
        Ted Yu added a comment - The test failure was due to HBASE-5936 . I updated workspace and it passed. Let's see what Hadoop QA tells us: https://builds.apache.org/job/PreCommit-HBASE-Build/2062/console
        Hide
        stack added a comment -

        @Ted np Sorry about that. I'll go ahead and commit this.

        Show
        stack added a comment - @Ted np Sorry about that. I'll go ahead and commit this.
        Hide
        stack added a comment -

        Committed to trunk. Nice feature. Thanks Elliott. Add a release note please.

        Show
        stack added a comment - Committed to trunk. Nice feature. Thanks Elliott. Add a release note please.
        Hide
        Elliott Clark added a comment -

        done

        Show
        Elliott Clark added a comment - done
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12530263/HBASE-5959-14.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 16 new or modified tests.

        +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2062//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2062//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12530263/HBASE-5959-14.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 16 new or modified tests. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2062//testReport/ Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2062//console This message is automatically generated.
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK #2957 (See https://builds.apache.org/job/HBase-TRUNK/2957/)
        HBASE-5959 Add other load balancers (Revision 1344457)

        Result = FAILURE
        stack :
        Files :

        • /hbase/trunk/hbase-server/pom.xml
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/DefaultLoadBalancer.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/LoadBalancer.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/LoadBalancerFactory.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionPlan.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerAndLoad.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/ClusterLoadState.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/DefaultLoadBalancer.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/LoadBalancerFactory.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/RegionInfoComparator.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/RegionLocationFinder.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/ServerAndLoad.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/TestRegionRebalancing.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestDefaultLoadBalancer.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/BalancerTestBase.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestBaseLoadBalancer.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestDefaultLoadBalancer.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestStochasticLoadBalancer.java
        • /hbase/trunk/pom.xml
        Show
        Hudson added a comment - Integrated in HBase-TRUNK #2957 (See https://builds.apache.org/job/HBase-TRUNK/2957/ ) HBASE-5959 Add other load balancers (Revision 1344457) Result = FAILURE stack : Files : /hbase/trunk/hbase-server/pom.xml /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/DefaultLoadBalancer.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/LoadBalancer.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/LoadBalancerFactory.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionPlan.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerAndLoad.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/ClusterLoadState.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/DefaultLoadBalancer.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/LoadBalancerFactory.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/RegionInfoComparator.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/RegionLocationFinder.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/ServerAndLoad.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/TestRegionRebalancing.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestDefaultLoadBalancer.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/BalancerTestBase.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestBaseLoadBalancer.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestDefaultLoadBalancer.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestStochasticLoadBalancer.java /hbase/trunk/pom.xml
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #33 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/33/)
        HBASE-5959 Add other load balancers (Revision 1344457)

        Result = FAILURE
        stack :
        Files :

        • /hbase/trunk/hbase-server/pom.xml
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/DefaultLoadBalancer.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/LoadBalancer.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/LoadBalancerFactory.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionPlan.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerAndLoad.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/ClusterLoadState.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/DefaultLoadBalancer.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/LoadBalancerFactory.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/RegionInfoComparator.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/RegionLocationFinder.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/ServerAndLoad.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/TestRegionRebalancing.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestDefaultLoadBalancer.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/BalancerTestBase.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestBaseLoadBalancer.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestDefaultLoadBalancer.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestStochasticLoadBalancer.java
        • /hbase/trunk/pom.xml
        Show
        Hudson added a comment - Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #33 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/33/ ) HBASE-5959 Add other load balancers (Revision 1344457) Result = FAILURE stack : Files : /hbase/trunk/hbase-server/pom.xml /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/DefaultLoadBalancer.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/LoadBalancer.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/LoadBalancerFactory.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionPlan.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerAndLoad.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/ClusterLoadState.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/DefaultLoadBalancer.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/LoadBalancerFactory.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/RegionInfoComparator.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/RegionLocationFinder.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/ServerAndLoad.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/TestRegionRebalancing.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestDefaultLoadBalancer.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/BalancerTestBase.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestBaseLoadBalancer.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestDefaultLoadBalancer.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestStochasticLoadBalancer.java /hbase/trunk/pom.xml
        Hide
        Phabricator added a comment -

        eclark has abandoned the revision "HBASE-5959 [jira] Add other load balancers".

        REVISION DETAIL
        https://reviews.facebook.net/D3189

        To: JIRA, eclark
        Cc: tedyu, lhofhansl

        Show
        Phabricator added a comment - eclark has abandoned the revision " HBASE-5959 [jira] Add other load balancers". REVISION DETAIL https://reviews.facebook.net/D3189 To: JIRA, eclark Cc: tedyu, lhofhansl

          People

          • Assignee:
            Elliott Clark
            Reporter:
            Elliott Clark
          • Votes:
            0 Vote for this issue
            Watchers:
            12 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development