Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-14611

Backport HBASE-14473 (Compute region locality in parallel) to 0.98

Details

    • Task
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.98.16
    • None
    • None
    • Reviewed

    Description

      eclark contributed a nice change on HBASE-14473 that scales calculation of locality balance cost to larger clusters. I'd like to bring this back to 0.98 for folks running it on larger clusters. The changes require a partial rewrite of RegionLocationFinder hence a backport issue.

      The difference between this backport and RegionLocationFinder on later branches is we preserve the ability to change the expiration time of cache items with the configuration parameter "hbase.master.balancer.regionLocationCacheTime".

      The difference between RegionLocationFinder in 0.98 before and after this change is the expiration time of cache entries will be set according to when they are written into the cache instead of from when they are last accessed, which seems better to me.

      Attachments

        1. HBASE-14611-0.98.patch
          11 kB
          Andrew Kyle Purtell

        Activity

          Attaching patch for precommit.

          Local results:

          -------------------------------------------------------
           T E S T S
          -------------------------------------------------------
          Running org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancer
          Tests run: 13, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 79.557 sec - in org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancer
          Running org.apache.hadoop.hbase.master.balancer.TestBaseLoadBalancer
          Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.193 sec - in org.apache.hadoop.hbase.master.balancer.TestBaseLoadBalancer
          Running org.apache.hadoop.hbase.master.balancer.TestDefaultLoadBalancer
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.858 sec - in org.apache.hadoop.hbase.master.balancer.TestDefaultLoadBalancer
          Running org.apache.hadoop.hbase.master.TestMasterFailoverBalancerPersistence
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.614 sec - in org.apache.hadoop.hbase.master.TestMasterFailoverBalancerPersistence
          
          Results :
          
          Tests run: 20, Failures: 0, Errors: 0, Skipped: 0
          
          apurtell Andrew Kyle Purtell added a comment - Attaching patch for precommit. Local results: ------------------------------------------------------- T E S T S ------------------------------------------------------- Running org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancer Tests run: 13, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 79.557 sec - in org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancer Running org.apache.hadoop.hbase.master.balancer.TestBaseLoadBalancer Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.193 sec - in org.apache.hadoop.hbase.master.balancer.TestBaseLoadBalancer Running org.apache.hadoop.hbase.master.balancer.TestDefaultLoadBalancer Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.858 sec - in org.apache.hadoop.hbase.master.balancer.TestDefaultLoadBalancer Running org.apache.hadoop.hbase.master.TestMasterFailoverBalancerPersistence Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.614 sec - in org.apache.hadoop.hbase.master.TestMasterFailoverBalancerPersistence Results : Tests run: 20, Failures: 0, Errors: 0, Skipped: 0
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12766727/HBASE-14611-0.98.patch
          against 0.98 branch at commit 46f2877381afefd1b7c838e10a80776b043854ae.
          ATTACHMENT ID: 12766727

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 hadoop versions. The patch compiles with all supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0 2.7.1)

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 protoc. The applied patch does not increase the total number of protoc compiler warnings.

          -1 javadoc. The javadoc tool appears to have generated 29 warning messages.

          +1 checkstyle. The applied patch does not increase the total number of checkstyle errors

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          +1 site. The mvn post-site goal succeeds with this patch.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/16024//testReport/
          Release Findbugs (version 2.0.3) warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/16024//artifact/patchprocess/newFindbugsWarnings.html
          Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/16024//artifact/patchprocess/checkstyle-aggregate.html

          Javadoc warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/16024//artifact/patchprocess/patchJavadocWarnings.txt
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/16024//console

          This message is automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12766727/HBASE-14611-0.98.patch against 0.98 branch at commit 46f2877381afefd1b7c838e10a80776b043854ae. ATTACHMENT ID: 12766727 +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop versions . The patch compiles with all supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0 2.7.1) +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 protoc . The applied patch does not increase the total number of protoc compiler warnings. -1 javadoc . The javadoc tool appears to have generated 29 warning messages. +1 checkstyle . The applied patch does not increase the total number of checkstyle errors +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn post-site goal succeeds with this patch. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/16024//testReport/ Release Findbugs (version 2.0.3) warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/16024//artifact/patchprocess/newFindbugsWarnings.html Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/16024//artifact/patchprocess/checkstyle-aggregate.html Javadoc warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/16024//artifact/patchprocess/patchJavadocWarnings.txt Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/16024//console This message is automatically generated.

          Any concerns about this change in 0.98 eclark?

          apurtell Andrew Kyle Purtell added a comment - Any concerns about this change in 0.98 eclark ?

          Any concerns lhofhansl ?

          apurtell Andrew Kyle Purtell added a comment - Any concerns lhofhansl ?
          yuzhihong@gmail.com Ted Yu added a comment -

          +1

          yuzhihong@gmail.com Ted Yu added a comment - +1

          No concerns, everything looks good to me.
          +1

          eclark Elliott Neil Clark added a comment - No concerns, everything looks good to me. +1

          Thanks for the reviews! Pushed to 0.98

          apurtell Andrew Kyle Purtell added a comment - Thanks for the reviews! Pushed to 0.98
          hudson Hudson added a comment -

          FAILURE: Integrated in HBase-0.98-matrix #258 (See https://builds.apache.org/job/HBase-0.98-matrix/258/)
          HBASE-14611 Backport HBASE-14473 (Compute region locality in parallel) (apurtell: rev 70b9015a0ec8de5ed9f9d9302b67b8e79bc4c558)

          • hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/RegionLocationFinder.java
          hudson Hudson added a comment - FAILURE: Integrated in HBase-0.98-matrix #258 (See https://builds.apache.org/job/HBase-0.98-matrix/258/ ) HBASE-14611 Backport HBASE-14473 (Compute region locality in parallel) (apurtell: rev 70b9015a0ec8de5ed9f9d9302b67b8e79bc4c558) hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/RegionLocationFinder.java
          hudson Hudson added a comment -

          FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #1131 (See https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/1131/)
          HBASE-14611 Backport HBASE-14473 (Compute region locality in parallel) (apurtell: rev 70b9015a0ec8de5ed9f9d9302b67b8e79bc4c558)

          • hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/RegionLocationFinder.java
          hudson Hudson added a comment - FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #1131 (See https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/1131/ ) HBASE-14611 Backport HBASE-14473 (Compute region locality in parallel) (apurtell: rev 70b9015a0ec8de5ed9f9d9302b67b8e79bc4c558) hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/RegionLocationFinder.java
          larsfrancke Lars Francke added a comment -

          This issue was closed as part of a bulk closing operation on 2015-11-20. All issues that have been resolved and where all fixVersions have been released have been closed (following discussions on the mailing list).

          larsfrancke Lars Francke added a comment - This issue was closed as part of a bulk closing operation on 2015-11-20. All issues that have been resolved and where all fixVersions have been released have been closed (following discussions on the mailing list).
          linbao111 linbao111 added a comment -

          i supect that something wrong with following patch code:

          • cache = createCache(conf.getInt("hbase.master.balancer.regionLocationCacheTime", 30));
            + // Preserve this tunable in 0.98
            + this.cacheTime = TimeUnit.SECONDS.toMillis(
            + conf.getInt("hbase.master.balancer.regionLocationCacheTime", 30));

          before patch,hbase.master.balancer.regionLocationCacheTime default 30minutes
          after patch hbase.master.balancer.regionLocationCacheTime default 30 seconds

          am i right?

          linbao111 linbao111 added a comment - i supect that something wrong with following patch code: cache = createCache(conf.getInt("hbase.master.balancer.regionLocationCacheTime", 30)); + // Preserve this tunable in 0.98 + this.cacheTime = TimeUnit.SECONDS.toMillis( + conf.getInt("hbase.master.balancer.regionLocationCacheTime", 30)); before patch,hbase.master.balancer.regionLocationCacheTime default 30minutes after patch hbase.master.balancer.regionLocationCacheTime default 30 seconds am i right?
          linbao111 linbao111 added a comment -

          more to be added,i think hbase.master.balancer.regionLocationCacheTime default value after patch did a serious impact on hdfs
          namenode(hdfs-audit.log become huge,and full of hmaster accessing)

          so if any one update to hbase 0.98.16.1,rememer to add hbase.master.balancer.regionLocationCacheTime to be 30*2*60 in hbase-site.xml,

          linbao111 linbao111 added a comment - more to be added,i think hbase.master.balancer.regionLocationCacheTime default value after patch did a serious impact on hdfs namenode(hdfs-audit.log become huge,and full of hmaster accessing) so if any one update to hbase 0.98.16.1,rememer to add hbase.master.balancer.regionLocationCacheTime to be 30*2*60 in hbase-site.xml,

          People

            apurtell Andrew Kyle Purtell
            apurtell Andrew Kyle Purtell
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: