Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Later
    • Affects Version/s: 0.90.2, 0.90.3, 0.90.4, 0.92.0
    • Fix Version/s: None
    • Component/s: master, regionserver
    • Labels:
      None
    • Release Note:
      Hide
      Patch used for table priority alone,In this stand alone patch, not only tables can have different priorities but also the different actions like "get","scan","put" and "delete" can have priorities.
      Show
      Patch used for table priority alone,In this stand alone patch, not only tables can have different priorities but also the different actions like "get","scan","put" and "delete" can have priorities.

      Description

      The HBase isolation and allocation tool is designed to help users manage cluster resource among different application and tables.
      When we have a large scale of HBase cluster with many applications running on it, there will be lots of problems. In Taobao there is a cluster for many departments to test their applications performance, these applications are based on HBase. With one cluster which has 12 servers, there will be only one application running exclusively on this server, and many other applications must wait until the previous test finished.
      After we add allocation manage function to the cluster, applications can share the cluster and run concurrently. Also if the Test Engineer wants to make sure there is no interference, he/she can move out other tables from this group.
      In groups we use table priority to allocate resource, when system is busy; we can make sure high-priority tables are not affected lower-priority tables
      Different groups can have different region server configurations, some groups optimized for reading can have large block cache size, and others optimized for writing can have large memstore size.
      Tables and region servers can be moved easily between groups; after changing the configuration, a group can be restarted alone instead of restarting the whole cluster.

      git entry : https://github.com/ICT-Ope/HBase_allocation .

      We hope our work is helpful.

      1. Performance_of_Table_priority.pdf
        486 kB
        Liu Jia
      2. HBase_isolation_and_allocation_user_guide.pdf
        1.77 MB
        Liu Jia
      3. Design_document_for_HBase_isolation_and_allocation.pdf
        140 kB
        Liu Jia
      4. System Structure.jpg
        61 kB
        Liu Jia
      5. Design_document_for_HBase_isolation_and_allocation_Revised.pdf
        558 kB
        Liu Jia
      6. TablePriority.patch
        37 kB
        Liu Jia
      7. TablePriority_v8_for_trunk.patch
        74 kB
        Liu Jia
      8. TablePriority_v8.patch
        74 kB
        Ted Yu
      9. TablePriority_v8.patch
        76 kB
        Liu Jia
      10. TablePrioriy_v9.patch
        76 kB
        Liu Jia
      11. TablePriority_v12.patch
        78 kB
        Liu Jia
      12. TablePriority_v12.patch
        78 kB
        Liu Jia
      13. TablePriority_v15_with_coprocessor.patch
        79 kB
        Liu Jia
      14. TablePriority_v16_with_coprocessor.patch
        82 kB
        Liu Jia
      15. Simple_YCSB_Tests_For_TablePriority_Trunk_and_0.90.4.pdf
        494 kB
        Liu Jia
      16. TablePriority_v17.patch
        116 kB
        Liu Jia
      17. TablePriority_v17.patch
        116 kB
        Liu Jia

        Issue Links

          Activity

          Liu Jia created issue -
          Liu Jia made changes -
          Field Original Value New Value
          Attachment Performance_of_Table_priority.pdf [ 12487422 ]
          Liu Jia made changes -
          Liu Jia made changes -
          Liu Jia made changes -
          Attachment System Structure.jpg [ 12487671 ]
          Liu Jia made changes -
          Liu Jia made changes -
          Attachment TablePriority.patch [ 12489340 ]
          Liu Jia made changes -
          Affects Version/s 0.90.3 [ 12316313 ]
          Affects Version/s 0.90.4 [ 12316406 ]
          Affects Version/s 0.92.0 [ 12314223 ]
          Liu Jia made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Release Note Patch used for table priority alone,In this stand alone patch, not only tables can have different priorities but also the different actions like "get","scan","put" and "delete" can have priorities.
          Fix Version/s 0.90.3 [ 12316313 ]
          stack made changes -
          Fix Version/s 0.90.5 [ 12317145 ]
          Fix Version/s 0.90.3 [ 12316313 ]
          Jean-Daniel Cryans made changes -
          Fix Version/s 0.94.0 [ 12316419 ]
          Fix Version/s 0.90.5 [ 12317145 ]
          Liu Jia made changes -
          Assignee Liu Jia [ liujia_ict ]
          Liu Jia made changes -
          Attachment TablePriority_v8_for_trunk.patch [ 12502140 ]
          Ted Yu made changes -
          Attachment TablePriority_v8.patch [ 12504702 ]
          Ted Yu made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Ted Yu made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Liu Jia made changes -
          Attachment TablePriority_v8.patch [ 12504706 ]
          Liu Jia made changes -
          Attachment TablePrioriy_v9.patch [ 12505326 ]
          Ted Yu made changes -
          Comment [ -1 overall. Here are the results of testing the latest attachment
            http://issues.apache.org/jira/secure/attachment/12505326/TablePrioriy_v9.patch
            against trunk revision .

              +1 @author. The patch does not contain any @author tags.

              +1 tests included. The patch appears to include 9 new or modified tests.

              -1 javadoc. The javadoc tool appears to have generated -140 warning messages.

              +1 javac. The applied patch does not increase the total number of javac compiler warnings.

              -1 findbugs. The patch appears to introduce 84 new Findbugs (version 1.3.9) warnings.

              +1 release audit. The applied patch does not increase the total number of release audit warnings.

               -1 core tests. The patch failed these unit tests:
                                 org.apache.hadoop.hbase.client.TestAdmin

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/393//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/393//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/393//console

          This message is automatically generated. ]
          Liu Jia made changes -
          Attachment TablePriority_v12.patch [ 12506220 ]
          Liu Jia made changes -
          Attachment TablePriority_v12.patch [ 12506221 ]
          Ted Yu made changes -
          Comment [ -1 overall. Here are the results of testing the latest attachment
            http://issues.apache.org/jira/secure/attachment/12502140/TablePriority_v8_for_trunk.patch
            against trunk revision .

              +1 @author. The patch does not contain any @author tags.

              +1 tests included. The patch appears to include 9 new or modified tests.

              -1 javadoc. The javadoc tool appears to have generated -145 warning messages.

              +1 javac. The applied patch does not increase the total number of javac compiler warnings.

              -1 findbugs. The patch appears to introduce 68 new Findbugs (version 1.3.9) warnings.

              +1 release audit. The applied patch does not increase the total number of release audit warnings.

               -1 core tests. The patch failed these unit tests:
                                 org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary
                            org.apache.hadoop.hbase.master.TestDistributedLogSplitting

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/154//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/154//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/154//console

          This message is automatically generated. ]
          Ted Yu made changes -
          Comment [ -1 overall. Here are the results of testing the latest attachment
            http://issues.apache.org/jira/secure/attachment/12504702/TablePriority_v8.patch
            against trunk revision .

              +1 @author. The patch does not contain any @author tags.

              +1 tests included. The patch appears to include 9 new or modified tests.

              -1 javadoc. The javadoc tool appears to have generated -143 warning messages.

              +1 javac. The applied patch does not increase the total number of javac compiler warnings.

              -1 findbugs. The patch appears to introduce 88 new Findbugs (version 1.3.9) warnings.

              +1 release audit. The applied patch does not increase the total number of release audit warnings.

               -1 core tests. The patch failed these unit tests:
                                 org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildOverlap
                            org.apache.hadoop.hbase.master.TestMasterFailover
                            org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildHole
                            org.apache.hadoop.hbase.client.TestAdmin
                            org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildBase
                            org.apache.hadoop.hbase.master.TestRestartCluster

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/328//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/328//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/328//console

          This message is automatically generated. ]
          Ted Yu made changes -
          Comment [ -1 overall. Here are the results of testing the latest attachment
            http://issues.apache.org/jira/secure/attachment/12504706/TablePriority_v8.patch
            against trunk revision .

              +1 @author. The patch does not contain any @author tags.

              +1 tests included. The patch appears to include 9 new or modified tests.

              -1 javadoc. The javadoc tool appears to have generated -143 warning messages.

              +1 javac. The applied patch does not increase the total number of javac compiler warnings.

              -1 findbugs. The patch appears to introduce 88 new Findbugs (version 1.3.9) warnings.

              +1 release audit. The applied patch does not increase the total number of release audit warnings.

               -1 core tests. The patch failed these unit tests:
                                 org.apache.hadoop.hbase.master.TestRestartCluster
                            org.apache.hadoop.hbase.client.TestAdmin
                            org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildBase
                            org.apache.hadoop.hbase.master.TestMasterFailover
                            org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildOverlap

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/330//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/330//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/330//console

          This message is automatically generated. ]
          Liu Jia made changes -
          Liu Jia made changes -
          Liu Jia made changes -
          Liu Jia made changes -
          Attachment TablePriority_v17.patch [ 12507887 ]
          Ted Yu made changes -
          Comment [ -1 overall. Here are the results of testing the latest attachment
            http://issues.apache.org/jira/secure/attachment/12506221/TablePriority_v12.patch
            against trunk revision .

              +1 @author. The patch does not contain any @author tags.

              +1 tests included. The patch appears to include 9 new or modified tests.

              -1 javadoc. The javadoc tool appears to have generated -140 warning messages.

              +1 javac. The applied patch does not increase the total number of javac compiler warnings.

              -1 findbugs. The patch appears to introduce 88 new Findbugs (version 1.3.9) warnings.

              +1 release audit. The applied patch does not increase the total number of release audit warnings.

              +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/452//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/452//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/452//console

          This message is automatically generated. ]
          Ted Yu made changes -
          Comment [ -1 overall. Here are the results of testing the latest attachment
            http://issues.apache.org/jira/secure/attachment/12507150/TablePriority_v15_with_coprocessor.patch
            against trunk revision .

              +1 @author. The patch does not contain any @author tags.

              +1 tests included. The patch appears to include 9 new or modified tests.

              -1 javadoc. The javadoc tool appears to have generated -128 warning messages.

              +1 javac. The applied patch does not increase the total number of javac compiler warnings.

              -1 findbugs. The patch appears to introduce 93 new Findbugs (version 1.3.9) warnings.

              +1 release audit. The applied patch does not increase the total number of release audit warnings.

               -1 core tests. The patch failed these unit tests:
                                 org.apache.hadoop.hbase.client.TestAdmin
                            org.apache.hadoop.hbase.client.TestInstantSchemaChange
                            org.apache.hadoop.hbase.io.hfile.TestHFileBlock

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/496//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/496//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/496//console

          This message is automatically generated. ]
          Ted Yu made changes -
          Comment [ -1 overall. Here are the results of testing the latest attachment
            http://issues.apache.org/jira/secure/attachment/12507382/Simple_YCSB_Tests_For_TablePriority_Trunk_and_0.90.4.pdf
            against trunk revision .

              +1 @author. The patch does not contain any @author tags.

              -1 tests included. The patch doesn't appear to include any new or modified tests.
                                  Please justify why no new tests are needed for this patch.
                                  Also please list what manual steps were performed to verify this patch.

              -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/507//console

          This message is automatically generated. ]
          Ted Yu made changes -
          Comment [ -1 overall. Here are the results of testing the latest attachment
            http://issues.apache.org/jira/secure/attachment/12507380/TablePriority_v16_with_coprocessor.patch
            against trunk revision .

              +1 @author. The patch does not contain any @author tags.

              +1 tests included. The patch appears to include 15 new or modified tests.

              -1 javadoc. The javadoc tool appears to have generated -128 warning messages.

              +1 javac. The applied patch does not increase the total number of javac compiler warnings.

              -1 findbugs. The patch appears to introduce 93 new Findbugs (version 1.3.9) warnings.

              +1 release audit. The applied patch does not increase the total number of release audit warnings.

               -1 core tests. The patch failed these unit tests:
                                 org.apache.hadoop.hbase.client.TestAdmin
                            org.apache.hadoop.hbase.client.TestInstantSchemaChange
                            org.apache.hadoop.hbase.coprocessor.TestClassLoading

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/505//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/505//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/505//console

          This message is automatically generated. ]
          Ted Yu made changes -
          Comment [
          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/1422/
          -----------------------------------------------------------

          Review request for hbase.


          Summary
          -------

          Test cases used for table priority.


          This addresses bug HBase-4120.
              https://issues.apache.org/jira/browse/HBase-4120


          Diffs
          -----

            http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/allocation/test/TestForActionPriority.java PRE-CREATION
            http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/allocation/test/TestForPriorityJobQueue.java PRE-CREATION
            http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/allocation/test/TestForTablePriority.java PRE-CREATION

          Diff: https://reviews.apache.org/r/1422/diff


          Testing
          -------


          Thanks,

          Jia

          ]
          Ted Yu made changes -
          Comment [
          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/1421/
          -----------------------------------------------------------

          (Updated 2011-08-09 13:38:32.587600)


          Review request for hbase.


          Changes
          -------

          dos2unix formatted


          Summary
          -------

          Patch used for table priority alone,In this patch, not only tables can have different priorities but also the different actions like "get","scan","put" and "delete" can have priorities.


          This addresses bug HBase-4120.
              https://issues.apache.org/jira/browse/HBase-4120


          Diffs (updated)
          -----

            http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java 1155226
            http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 1155226
            http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/PriorityHBaseServer.java PRE-CREATION
            http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/PriorityJobQueue.java PRE-CREATION

          Diff: https://reviews.apache.org/r/1421/diff


          Testing
          -------

          Tested with test cases in TestCase_For_TablePriority_trunk_v1.patch
          please apply the patch of HBASE-4181 first,in some circumstances this bug will affect the performance of client.


          Thanks,

          Jia

          ]
          Ted Yu made changes -
          Comment [
          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/1421/
          -----------------------------------------------------------

          (Updated 2011-08-09 14:41:11.616392)


          Review request for hbase.


          Changes
          -------

          Add test cases.


          Summary (updated)
          -------

          Patch used for table priority alone,In this patch, not only tables can have different priorities but also the different actions like "get","scan","put" and "delete" can have priorities.


          This addresses bug HBase-4120.
              https://issues.apache.org/jira/browse/HBase-4120


          Diffs (updated)
          -----

            http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java 1155226
            http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 1155226
            http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/PriorityHBaseServer.java PRE-CREATION
            http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/PriorityJobQueue.java PRE-CREATION
            http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/allocation/test/TestForActionPriority.java PRE-CREATION
            http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/allocation/test/TestForPriorityJobQueue.java PRE-CREATION
            http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/allocation/test/TestForTablePriority.java PRE-CREATION

          Diff: https://reviews.apache.org/r/1421/diff


          Testing
          -------

          Tested with test cases in TestCase_For_TablePriority_trunk_v1.patch
          please apply the patch of HBASE-4181 first,in some circumstances this bug will affect the performance of client.


          Thanks,

          Jia

          ]
          Ted Yu made changes -
          Comment [
          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/1421/
          -----------------------------------------------------------

          (Updated 2011-10-26 14:13:10.677021)


          Review request for hbase.


          Changes
          -------

          Fix bugs


          Summary
          -------

          Patch used for table priority alone,In this patch, not only tables can have different priorities but also the different actions like "get","scan","put" and "delete" can have priorities.


          This addresses bug HBase-4120.
              https://issues.apache.org/jira/browse/HBase-4120


          Diffs (updated)
          -----

            http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java 1189169
            http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 1189169
            http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/PriorityFunction.java PRE-CREATION
            http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/PriorityHBaseServer.java PRE-CREATION
            http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/PriorityJobQueue.java PRE-CREATION
            http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1189169
            http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/allocation/test/TestForActionPriority.java PRE-CREATION
            http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/allocation/test/TestForPriorityJobQueue.java PRE-CREATION
            http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/allocation/test/TestForTablePriority.java PRE-CREATION

          Diff: https://reviews.apache.org/r/1421/diff


          Testing
          -------

          Tested with test cases in TestCase_For_TablePriority_trunk_v1.patch
          please apply the patch of HBASE-4181 first,in some circumstances this bug will affect the performance of client.


          Thanks,

          Jia

          ]
          Liu Jia made changes -
          Attachment TablePriority_v17.patch [ 12508188 ]
          Lars Hofhansl made changes -
          Fix Version/s 0.96.0 [ 12320040 ]
          Fix Version/s 0.94.0 [ 12316419 ]
          Francis Liu made changes -
          Link This issue is related to HBASE-6721 [ HBASE-6721 ]
          stack made changes -
          Fix Version/s 0.96.0 [ 12320040 ]
          Andrew Purtell made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Assignee Liu Jia [ liujia_ict ]
          Resolution Later [ 7 ]

            People

            • Assignee:
              Unassigned
              Reporter:
              Liu Jia
            • Votes:
              3 Vote for this issue
              Watchers:
              38 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 504h
                504h
                Remaining:
                Remaining Estimate - 504h
                504h
                Logged:
                Time Spent - Not Specified
                Not Specified

                  Development