Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.0
    • Component/s: regionserver
    • Labels:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      [ADVANCED USERS ONLY] This patch adds a new experimental module hbase-rsgroup. It is an advanced feature for partitioning regionservers into distinctive groups for strict isolation, and should only be used by users who are sophisticated enough to understand the full implications and have a sufficient background in managing HBase clusters.

      RSGroups can be defined and managed with shell commands or corresponding Java APIs. A server can be added to a group with hostname and port pair, and tables can be moved to this group so that only regionservers in the same rsgroup can host the regions of the table. RegionServers and tables can only belong to 1 group at a time. By default, all tables and regionservers belong to the "default" group. System tables can also be put into a group using the regular APIs. A custom balancer implementation tracks assignments per rsgroup and makes sure to move regions to the relevant regionservers in that group. The group information is stored in a regular HBase table, and a zookeeper-based read-only cache is used at the cluster bootstrap time.

      To enable, add the following to your hbase-site.xml and restart your Master:


       <property>
         <name>hbase.coprocessor.master.classes</name>
         <value>org.apache.hadoop.hbase.rsgroup.RSGroupAdminEndpoint</value>
       </property>
       <property>
         <name>hbase.master.loadbalancer.class</name>
         <value>org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer</value>
       </property>


      Then use the shell 'rsgroup' commands to create and manipulate regionserver groups: e.g. to add a group and then add a server to it, do as follows:

       hbase(main):008:0> add_rsgroup 'my_group'
       Took 0.5610 seconds

      This adds a group to the 'hbase:rsgroup' system table. Add a server (hostname + port) to the group using the 'move_rsgroup_servers' command as follows:

       hbase(main):010:0> move_rsgroup_servers 'my_group',['k.att.net:51129']
      Show
      [ADVANCED USERS ONLY] This patch adds a new experimental module hbase-rsgroup. It is an advanced feature for partitioning regionservers into distinctive groups for strict isolation, and should only be used by users who are sophisticated enough to understand the full implications and have a sufficient background in managing HBase clusters. RSGroups can be defined and managed with shell commands or corresponding Java APIs. A server can be added to a group with hostname and port pair, and tables can be moved to this group so that only regionservers in the same rsgroup can host the regions of the table. RegionServers and tables can only belong to 1 group at a time. By default, all tables and regionservers belong to the "default" group. System tables can also be put into a group using the regular APIs. A custom balancer implementation tracks assignments per rsgroup and makes sure to move regions to the relevant regionservers in that group. The group information is stored in a regular HBase table, and a zookeeper-based read-only cache is used at the cluster bootstrap time. To enable, add the following to your hbase-site.xml and restart your Master:  <property>    <name>hbase.coprocessor.master.classes</name>    <value>org.apache.hadoop.hbase.rsgroup.RSGroupAdminEndpoint</value>  </property>  <property>    <name>hbase.master.loadbalancer.class</name>    <value>org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer</value>  </property> Then use the shell 'rsgroup' commands to create and manipulate regionserver groups: e.g. to add a group and then add a server to it, do as follows:  hbase(main):008:0> add_rsgroup 'my_group'  Took 0.5610 seconds This adds a group to the 'hbase:rsgroup' system table. Add a server (hostname + port) to the group using the 'move_rsgroup_servers' command as follows:  hbase(main):010:0> move_rsgroup_servers 'my_group',['k.att.net:51129']

      Description

      In multi-tenant deployments of HBase, it is likely that a RegionServer will be serving out regions from a number of different tables owned by various client applications. Being able to group a subset of running RegionServers and assign specific tables to it, provides a client application a level of isolation and resource allocation.

      The proposal essentially is to have an AssignmentManager which is aware of RegionServer groups and assigns tables to region servers based on groupings. Load balancing will occur on a per group basis as well.

      This is essentially a simplification of the approach taken in HBASE-4120. See attached document.

      1. HBASE-6721-DesigDoc.pdf
        171 kB
        Vandana Ayyalasomayajula
      2. HBASE-6721_94.patch
        117 kB
        Francis Liu
      3. HBASE-6721_94.patch
        162 kB
        Francis Liu
      4. HBASE-6721_94_2.patch
        150 kB
        Francis Liu
      5. HBASE-6721_94_3.patch
        352 kB
        Francis Liu
      6. HBASE-6721_94_3.patch
        131 kB
        Francis Liu
      7. HBASE-6721_94_4.patch
        180 kB
        Francis Liu
      8. HBASE-6721_94_5.patch
        184 kB
        Francis Liu
      9. HBASE-6721_94_6.patch
        196 kB
        Francis Liu
      10. HBASE-6721_94_7.patch
        201 kB
        Francis Liu
      11. HBASE-6721_trunk.patch
        225 kB
        Francis Liu
      12. HBASE-6721_trunk.patch
        225 kB
        Francis Liu
      13. HBASE-6721_trunk.patch
        771 kB
        Francis Liu
      14. HBASE-6721-DesigDoc.pdf
        86 kB
        Francis Liu
      15. HBASE-6721-DesigDoc.pdf
        89 kB
        Francis Liu
      16. 6721-master-webUI.patch
        3 kB
        chunhui shen
      17. HBASE-6721_8.patch
        778 kB
        Francis Liu
      18. HBASE-6721-DesigDoc.pdf
        91 kB
        Francis Liu
      19. HBASE-6721_9.patch
        775 kB
        Francis Liu
      20. HBASE-6721_9.patch
        779 kB
        Francis Liu
      21. HBASE-6721_10.patch
        781 kB
        Francis Liu
      22. HBASE-6721_trunk1.patch
        820 kB
        Francis Liu
      23. HBASE-6721_trunk2.patch
        820 kB
        Francis Liu
      24. HBASE-6721_11.patch
        942 kB
        Francis Liu
      25. balanceCluster Sequence Diagram.svg
        81 kB
        Guorui Wu
      26. HBASE-6721 GroupBasedLoadBalancer Sequence Diagram.xml
        27 kB
        Guorui Wu
      27. immediateAssignments Sequence Diagram.svg
        94 kB
        Guorui Wu
      28. randomAssignment Sequence Diagram.svg
        90 kB
        Guorui Wu
      29. retainAssignment Sequence Diagram.svg
        150 kB
        Guorui Wu
      30. roundRobinAssignment Sequence Diagram.svg
        95 kB
        Guorui Wu
      31. HBASE-6721_98_1.patch
        941 kB
        Francis Liu
      32. HBASE-6721_98_2.patch
        876 kB
        Francis Liu
      33. HBASE-6721_0.98_2.patch
        876 kB
        Nick Dimiduk
      34. HBASE-6721_hbase-6721_addendum.patch
        161 kB
        Francis Liu
      35. HBASE-6721_12.patch
        877 kB
        Francis Liu
      36. HBASE-6721_13.patch
        826 kB
        Francis Liu
      37. HBASE-6721_14.patch
        812 kB
        Francis Liu
      38. HBASE-6721_15.patch
        838 kB
        Francis Liu
      39. hbase-6721-v15-branch-1.1.patch
        848 kB
        Enis Soztutar
      40. hbase-6721-v16.patch
        840 kB
        Enis Soztutar
      41. hbase-6721-v17.patch
        841 kB
        Enis Soztutar
      42. hbase-6721-v18.patch
        856 kB
        Francis Liu
      43. hbase-6721-v19.patch
        857 kB
        Francis Liu
      44. hbase-6721-v20.patch
        859 kB
        Francis Liu
      45. hbase-6721-v21.patch
        860 kB
        Francis Liu
      46. hbase-6721-v22.patch
        861 kB
        Francis Liu
      47. hbase-6721-v23.patch
        863 kB
        Francis Liu
      48. hbase-6721-v25.patch
        863 kB
        Francis Liu
      49. hbase-6721-v26_draft1.patch
        854 kB
        Francis Liu
      50. hbase-6721-v26.patch
        853 kB
        Francis Liu
      51. hbase-6721-v27.patch
        853 kB
        Francis Liu
      52. hbase-6721-v27.patch.txt
        853 kB
        Enis Soztutar
      53. hbase-6721-v27.patch
        853 kB
        Enis Soztutar
      54. hbase-6721-v28.patch
        852 kB
        Enis Soztutar
      55. hbase-6721-v28.patch
        852 kB
        Francis Liu
      56. hbase-6721-v29.patch
        852 kB
        Francis Liu

        Issue Links

        There are no Sub-Tasks for this issue.

          Activity

          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Jenkins build HBase-1.5 #115 (See https://builds.apache.org/job/HBase-1.5/115/)
          Amend HBASE-15631 Backport Regionserver Groups (HBASE-6721) to branch-1 (apurtell: rev 089acc74dad0bd6368e4bb584fde2a84b8c8ab56)

          • (edit) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java
          • (edit) hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/master/balancer/TestRSGroupBasedLoadBalancer.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Jenkins build HBase-1.5 #115 (See https://builds.apache.org/job/HBase-1.5/115/ ) Amend HBASE-15631 Backport Regionserver Groups ( HBASE-6721 ) to branch-1 (apurtell: rev 089acc74dad0bd6368e4bb584fde2a84b8c8ab56) (edit) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java (edit) hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/master/balancer/TestRSGroupBasedLoadBalancer.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Jenkins build HBase-1.4 #972 (See https://builds.apache.org/job/HBase-1.4/972/)
          Amend HBASE-15631 Backport Regionserver Groups (HBASE-6721) to branch-1 (apurtell: rev 11b7e1bc8cafca686d020b67239248f1347ad775)

          • (edit) hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/master/balancer/TestRSGroupBasedLoadBalancer.java
          • (edit) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Jenkins build HBase-1.4 #972 (See https://builds.apache.org/job/HBase-1.4/972/ ) Amend HBASE-15631 Backport Regionserver Groups ( HBASE-6721 ) to branch-1 (apurtell: rev 11b7e1bc8cafca686d020b67239248f1347ad775) (edit) hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/master/balancer/TestRSGroupBasedLoadBalancer.java (edit) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Jenkins build HBase-1.5 #111 (See https://builds.apache.org/job/HBase-1.5/111/)
          Amend HBASE-15631 Backport Regionserver Groups (HBASE-6721) to branch-1 (apurtell: rev d0629d0f121e66a7d52caceb4ac290e879ba6eb1)

          • (edit) hbase-rsgroup/pom.xml
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Jenkins build HBase-1.5 #111 (See https://builds.apache.org/job/HBase-1.5/111/ ) Amend HBASE-15631 Backport Regionserver Groups ( HBASE-6721 ) to branch-1 (apurtell: rev d0629d0f121e66a7d52caceb4ac290e879ba6eb1) (edit) hbase-rsgroup/pom.xml
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build HBase-1.4 #967 (See https://builds.apache.org/job/HBase-1.4/967/)
          HBASE-15631 Backport Regionserver Groups (HBASE-6721) to branch-1 (apurtell: rev c320007638fe57f858aeac974cdd6bbd6b9dd5eb)

          • (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseMasterObserver.java
          • (add) hbase-protocol/src/main/protobuf/RSGroupAdmin.proto
          • (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterServices.java
          • (add) hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/VerifyingRSGroupAdminClient.java
          • (add) hbase-shell/src/main/ruby/shell/commands/move_tables_rsgroup.rb
          • (add) hbase-shell/src/main/ruby/shell/commands/get_table_rsgroup.rb
          • (add) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManager.java
          • (add) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminClient.java
          • (add) hbase-shell/src/main/ruby/shell/commands/add_rsgroup.rb
          • (add) hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsOfflineMode.java
          • (add) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManagerImpl.java
          • (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/LoadBalancer.java
          • (edit) hbase-shell/src/test/ruby/test_helper.rb
          • (add) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfo.java
          • (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/normalizer/TestSimpleRegionNormalizer.java
          • (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseMasterAndRegionObserver.java
          • (edit) hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.jamon
          • (add) hbase-shell/src/main/ruby/shell/commands/move_servers_tables_rsgroup.rb
          • (add) hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroups.java
          • (edit) hbase-shell/src/test/java/org/apache/hadoop/hbase/client/TestShell.java
          • (edit) pom.xml
          • (edit) hbase-shell/src/main/ruby/hbase/hbase.rb
          • (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java
          • (add) hbase-shell/src/main/ruby/shell/commands/move_servers_rsgroup.rb
          • (add) hbase-it/src/test/rsgroup/org/apache/hadoop/hbase/rsgroup/IntegrationTestRSGroup.java
          • (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/MasterObserver.java
          • (add) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupSerDe.java
          • (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/LocalHBaseCluster.java
          • (edit) hbase-shell/src/main/ruby/hbase.rb
          • (add) hbase-shell/src/main/ruby/shell/commands/get_server_rsgroup.rb
          • (edit) hbase-it/pom.xml
          • (add) hbase-protocol/src/main/protobuf/RSGroup.proto
          • (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterStatusServlet.java
          • (add) hbase-shell/src/main/ruby/hbase/rsgroup_admin.rb
          • (add) hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RSGroupProtos.java
          • (edit) hbase-shell/src/main/ruby/shell.rb
          • (add) hbase-shell/src/main/ruby/shell/commands/remove_rsgroup.rb
          • (add) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminEndpoint.java
          • (add) hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsBase.java
          • (add) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupProtobufUtil.java
          • (add) hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/master/balancer/TestRSGroupBasedLoadBalancer.java
          • (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController.java
          • (edit) hbase-protocol/pom.xml
          • (edit) hbase-shell/pom.xml
          • (add) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminServer.java
          • (add) hbase-common/src/main/java/org/apache/hadoop/hbase/net/Address.java
          • (add) hbase-shell/src/test/rsgroup/org/apache/hadoop/hbase/client/rsgroup/TestShellRSGroups.java
          • (edit) hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClientProtos.java
          • (edit) hbase-shell/src/main/ruby/shell/commands.rb
          • (add) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdmin.java
          • (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManagerOnCluster.java
          • (add) hbase-shell/src/main/ruby/shell/commands/get_rsgroup.rb
          • (add) hbase-rsgroup/pom.xml
          • (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/ServerName.java
          • (add) hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RSGroupAdminProtos.java
          • (add) hbase-shell/src/main/ruby/shell/commands/balance_rsgroup.rb
          • (add) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupableBalancer.java
          • (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterCoprocessorHost.java
          • (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
          • (add) hbase-shell/src/main/ruby/shell/commands/list_rsgroups.rb
          • (add) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java
          • (add) hbase-shell/src/test/ruby/shell/rsgroup_shell_test.rb
          • (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockNoopMasterServices.java
          • (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java
          • (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
          • (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build HBase-1.4 #967 (See https://builds.apache.org/job/HBase-1.4/967/ ) HBASE-15631 Backport Regionserver Groups ( HBASE-6721 ) to branch-1 (apurtell: rev c320007638fe57f858aeac974cdd6bbd6b9dd5eb) (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseMasterObserver.java (add) hbase-protocol/src/main/protobuf/RSGroupAdmin.proto (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterServices.java (add) hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/VerifyingRSGroupAdminClient.java (add) hbase-shell/src/main/ruby/shell/commands/move_tables_rsgroup.rb (add) hbase-shell/src/main/ruby/shell/commands/get_table_rsgroup.rb (add) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManager.java (add) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminClient.java (add) hbase-shell/src/main/ruby/shell/commands/add_rsgroup.rb (add) hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsOfflineMode.java (add) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManagerImpl.java (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/LoadBalancer.java (edit) hbase-shell/src/test/ruby/test_helper.rb (add) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfo.java (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/normalizer/TestSimpleRegionNormalizer.java (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseMasterAndRegionObserver.java (edit) hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.jamon (add) hbase-shell/src/main/ruby/shell/commands/move_servers_tables_rsgroup.rb (add) hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroups.java (edit) hbase-shell/src/test/java/org/apache/hadoop/hbase/client/TestShell.java (edit) pom.xml (edit) hbase-shell/src/main/ruby/hbase/hbase.rb (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java (add) hbase-shell/src/main/ruby/shell/commands/move_servers_rsgroup.rb (add) hbase-it/src/test/rsgroup/org/apache/hadoop/hbase/rsgroup/IntegrationTestRSGroup.java (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/MasterObserver.java (add) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupSerDe.java (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/LocalHBaseCluster.java (edit) hbase-shell/src/main/ruby/hbase.rb (add) hbase-shell/src/main/ruby/shell/commands/get_server_rsgroup.rb (edit) hbase-it/pom.xml (add) hbase-protocol/src/main/protobuf/RSGroup.proto (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterStatusServlet.java (add) hbase-shell/src/main/ruby/hbase/rsgroup_admin.rb (add) hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RSGroupProtos.java (edit) hbase-shell/src/main/ruby/shell.rb (add) hbase-shell/src/main/ruby/shell/commands/remove_rsgroup.rb (add) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminEndpoint.java (add) hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsBase.java (add) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupProtobufUtil.java (add) hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/master/balancer/TestRSGroupBasedLoadBalancer.java (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController.java (edit) hbase-protocol/pom.xml (edit) hbase-shell/pom.xml (add) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminServer.java (add) hbase-common/src/main/java/org/apache/hadoop/hbase/net/Address.java (add) hbase-shell/src/test/rsgroup/org/apache/hadoop/hbase/client/rsgroup/TestShellRSGroups.java (edit) hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClientProtos.java (edit) hbase-shell/src/main/ruby/shell/commands.rb (add) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdmin.java (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManagerOnCluster.java (add) hbase-shell/src/main/ruby/shell/commands/get_rsgroup.rb (add) hbase-rsgroup/pom.xml (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/ServerName.java (add) hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RSGroupAdminProtos.java (add) hbase-shell/src/main/ruby/shell/commands/balance_rsgroup.rb (add) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupableBalancer.java (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterCoprocessorHost.java (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java (add) hbase-shell/src/main/ruby/shell/commands/list_rsgroups.rb (add) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java (add) hbase-shell/src/test/ruby/shell/rsgroup_shell_test.rb (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockNoopMasterServices.java (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Jenkins build HBase-1.5 #110 (See https://builds.apache.org/job/HBase-1.5/110/)
          HBASE-15631 Backport Regionserver Groups (HBASE-6721) to branch-1 (apurtell: rev 64328caef0bb712bb69d0241b4b8b3474a82702c)

          • (edit) pom.xml
          • (edit) hbase-protocol/pom.xml
          • (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController.java
          • (add) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupProtobufUtil.java
          • (add) hbase-it/src/test/rsgroup/org/apache/hadoop/hbase/rsgroup/IntegrationTestRSGroup.java
          • (add) hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RSGroupProtos.java
          • (edit) hbase-shell/src/main/ruby/shell/commands.rb
          • (add) hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/master/balancer/TestRSGroupBasedLoadBalancer.java
          • (edit) hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClientProtos.java
          • (add) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfo.java
          • (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
          • (add) hbase-shell/src/main/ruby/shell/commands/balance_rsgroup.rb
          • (add) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupSerDe.java
          • (add) hbase-shell/src/main/ruby/shell/commands/move_tables_rsgroup.rb
          • (edit) hbase-shell/src/test/java/org/apache/hadoop/hbase/client/TestShell.java
          • (add) hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsOfflineMode.java
          • (add) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManagerImpl.java
          • (edit) hbase-shell/src/main/ruby/hbase.rb
          • (edit) hbase-shell/src/main/ruby/hbase/hbase.rb
          • (add) hbase-protocol/src/main/protobuf/RSGroup.proto
          • (add) hbase-shell/src/test/ruby/shell/rsgroup_shell_test.rb
          • (add) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupableBalancer.java
          • (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
          • (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterServices.java
          • (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterCoprocessorHost.java
          • (add) hbase-shell/src/main/ruby/shell/commands/get_server_rsgroup.rb
          • (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/LoadBalancer.java
          • (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/normalizer/TestSimpleRegionNormalizer.java
          • (add) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java
          • (add) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdmin.java
          • (add) hbase-shell/src/main/ruby/shell/commands/list_rsgroups.rb
          • (add) hbase-shell/src/main/ruby/shell/commands/move_servers_tables_rsgroup.rb
          • (edit) hbase-shell/pom.xml
          • (add) hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RSGroupAdminProtos.java
          • (add) hbase-shell/src/main/ruby/shell/commands/get_table_rsgroup.rb
          • (add) hbase-shell/src/main/ruby/hbase/rsgroup_admin.rb
          • (add) hbase-shell/src/test/rsgroup/org/apache/hadoop/hbase/client/rsgroup/TestShellRSGroups.java
          • (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java
          • (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
          • (edit) hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.jamon
          • (edit) hbase-it/pom.xml
          • (add) hbase-common/src/main/java/org/apache/hadoop/hbase/net/Address.java
          • (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseMasterObserver.java
          • (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java
          • (add) hbase-shell/src/main/ruby/shell/commands/remove_rsgroup.rb
          • (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManagerOnCluster.java
          • (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockNoopMasterServices.java
          • (edit) hbase-shell/src/main/ruby/shell.rb
          • (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/MasterObserver.java
          • (add) hbase-shell/src/main/ruby/shell/commands/move_servers_rsgroup.rb
          • (add) hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/VerifyingRSGroupAdminClient.java
          • (add) hbase-protocol/src/main/protobuf/RSGroupAdmin.proto
          • (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/ServerName.java
          • (add) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminEndpoint.java
          • (add) hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroups.java
          • (add) hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsBase.java
          • (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseMasterAndRegionObserver.java
          • (add) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManager.java
          • (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/LocalHBaseCluster.java
          • (add) hbase-shell/src/main/ruby/shell/commands/get_rsgroup.rb
          • (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterStatusServlet.java
          • (add) hbase-shell/src/main/ruby/shell/commands/add_rsgroup.rb
          • (edit) hbase-shell/src/test/ruby/test_helper.rb
          • (add) hbase-rsgroup/pom.xml
          • (add) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminClient.java
          • (add) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminServer.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Jenkins build HBase-1.5 #110 (See https://builds.apache.org/job/HBase-1.5/110/ ) HBASE-15631 Backport Regionserver Groups ( HBASE-6721 ) to branch-1 (apurtell: rev 64328caef0bb712bb69d0241b4b8b3474a82702c) (edit) pom.xml (edit) hbase-protocol/pom.xml (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController.java (add) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupProtobufUtil.java (add) hbase-it/src/test/rsgroup/org/apache/hadoop/hbase/rsgroup/IntegrationTestRSGroup.java (add) hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RSGroupProtos.java (edit) hbase-shell/src/main/ruby/shell/commands.rb (add) hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/master/balancer/TestRSGroupBasedLoadBalancer.java (edit) hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClientProtos.java (add) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfo.java (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java (add) hbase-shell/src/main/ruby/shell/commands/balance_rsgroup.rb (add) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupSerDe.java (add) hbase-shell/src/main/ruby/shell/commands/move_tables_rsgroup.rb (edit) hbase-shell/src/test/java/org/apache/hadoop/hbase/client/TestShell.java (add) hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsOfflineMode.java (add) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManagerImpl.java (edit) hbase-shell/src/main/ruby/hbase.rb (edit) hbase-shell/src/main/ruby/hbase/hbase.rb (add) hbase-protocol/src/main/protobuf/RSGroup.proto (add) hbase-shell/src/test/ruby/shell/rsgroup_shell_test.rb (add) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupableBalancer.java (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterServices.java (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterCoprocessorHost.java (add) hbase-shell/src/main/ruby/shell/commands/get_server_rsgroup.rb (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/LoadBalancer.java (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/normalizer/TestSimpleRegionNormalizer.java (add) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java (add) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdmin.java (add) hbase-shell/src/main/ruby/shell/commands/list_rsgroups.rb (add) hbase-shell/src/main/ruby/shell/commands/move_servers_tables_rsgroup.rb (edit) hbase-shell/pom.xml (add) hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RSGroupAdminProtos.java (add) hbase-shell/src/main/ruby/shell/commands/get_table_rsgroup.rb (add) hbase-shell/src/main/ruby/hbase/rsgroup_admin.rb (add) hbase-shell/src/test/rsgroup/org/apache/hadoop/hbase/client/rsgroup/TestShellRSGroups.java (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java (edit) hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.jamon (edit) hbase-it/pom.xml (add) hbase-common/src/main/java/org/apache/hadoop/hbase/net/Address.java (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseMasterObserver.java (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java (add) hbase-shell/src/main/ruby/shell/commands/remove_rsgroup.rb (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManagerOnCluster.java (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockNoopMasterServices.java (edit) hbase-shell/src/main/ruby/shell.rb (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/MasterObserver.java (add) hbase-shell/src/main/ruby/shell/commands/move_servers_rsgroup.rb (add) hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/VerifyingRSGroupAdminClient.java (add) hbase-protocol/src/main/protobuf/RSGroupAdmin.proto (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/ServerName.java (add) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminEndpoint.java (add) hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroups.java (add) hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsBase.java (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseMasterAndRegionObserver.java (add) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManager.java (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/LocalHBaseCluster.java (add) hbase-shell/src/main/ruby/shell/commands/get_rsgroup.rb (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterStatusServlet.java (add) hbase-shell/src/main/ruby/shell/commands/add_rsgroup.rb (edit) hbase-shell/src/test/ruby/test_helper.rb (add) hbase-rsgroup/pom.xml (add) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminClient.java (add) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminServer.java
          Hide
          stack stack added a comment -

          This issue is resolved chenxu Open a new one or ask up on the dev mailing list? Thanks.

          Show
          stack stack added a comment - This issue is resolved chenxu Open a new one or ask up on the dev mailing list? Thanks.
          Hide
          javaman_chen chenxu added a comment -
          RSGroupBasedLoadBalancer.roundRobinAssignment
            ...
            for(String groupKey : regionMap.keySet()) {
              if (regionMap.get(groupKey).size() > 0) {
                Map<ServerName, List<HRegionInfo>> result =
                this.internalBalancer.roundRobinAssignment(
                  regionMap.get(groupKey),
                  serverMap.get(groupKey));
                if(result != null) {
                  assignments.putAll(result);
                }
              }
            }
            ...
          

          if two group both has BOGUS_SERVER_NAME, assignments.putAll may be not right

          RSGroupBasedLoadBalancer.randomAssignment
          public ServerName randomAssignment(HRegionInfo region,
            List<ServerName> servers) throws HBaseIOException {
            ListMultimap<String,HRegionInfo> regionMap = LinkedListMultimap.create();
            ListMultimap<String,ServerName> serverMap = LinkedListMultimap.create();
            generateGroupMaps(Lists.newArrayList(region), servers, regionMap, serverMap);
            List<ServerName> filteredServers = serverMap.get(regionMap.keySet().iterator().next());
            return this.internalBalancer.randomAssignment(region, filteredServers);
          }
          

          if internalBalancer.randomAssignment return BOGUS_SERVER_NAME, how does AM work?
          I think we shoud return null instead of BOGUS_SERVER_NAME

          Show
          javaman_chen chenxu added a comment - RSGroupBasedLoadBalancer.roundRobinAssignment ... for ( String groupKey : regionMap.keySet()) { if (regionMap.get(groupKey).size() > 0) { Map<ServerName, List<HRegionInfo>> result = this .internalBalancer.roundRobinAssignment( regionMap.get(groupKey), serverMap.get(groupKey)); if (result != null ) { assignments.putAll(result); } } } ... if two group both has BOGUS_SERVER_NAME, assignments.putAll may be not right RSGroupBasedLoadBalancer.randomAssignment public ServerName randomAssignment(HRegionInfo region, List<ServerName> servers) throws HBaseIOException { ListMultimap< String ,HRegionInfo> regionMap = LinkedListMultimap.create(); ListMultimap< String ,ServerName> serverMap = LinkedListMultimap.create(); generateGroupMaps(Lists.newArrayList(region), servers, regionMap, serverMap); List<ServerName> filteredServers = serverMap.get(regionMap.keySet().iterator().next()); return this .internalBalancer.randomAssignment(region, filteredServers); } if internalBalancer.randomAssignment return BOGUS_SERVER_NAME, how does AM work? I think we shoud return null instead of BOGUS_SERVER_NAME
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #2505 (See https://builds.apache.org/job/HBase-Trunk_matrix/2505/)
          HBASE-17624 Address late review of HBASE-6721, rsgroups feature (stack: rev e019961150624236615fe7bd4a9017622d9f78d0)

          • (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/LoadBalancer.java
          • (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfo.java
          • (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java
          • (edit) hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroups.java
          • (edit) hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/master/balancer/TestRSGroupBasedLoadBalancer.java
          • (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java
          • (edit) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupSerDe.java
          • (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALSplit.java
          • (delete) hbase-common/src/test/java/org/apache/hadoop/hbase/util/TestAddressing.java
          • (edit) hbase-shell/src/main/ruby/shell/commands/get_table_rsgroup.rb
          • (edit) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java
          • (edit) hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestAsyncProcess.java
          • (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestSplitLogManager.java
          • (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
          • (edit) hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/VerifyingRSGroupAdminClient.java
          • (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterCoprocessorHost.java
          • (edit) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManager.java
          • (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestDurability.java
          • (edit) hbase-it/src/test/java/org/apache/hadoop/hbase/rsgroup/IntegrationTestRSGroup.java
          • (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAdmin2.java
          • (edit) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdmin.java
          • (edit) hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsOfflineMode.java
          • (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java
          • (add) hbase-common/src/main/java/org/apache/hadoop/hbase/util/Address.java
          • (edit) hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestSimpleRequestController.java
          • (edit) hbase-shell/src/main/ruby/shell/commands/balance_rsgroup.rb
          • (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/util/Addressing.java
          • (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java
          • (edit) hbase-shell/src/main/ruby/shell/commands/list_rsgroups.rb
          • (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/TestServerName.java
          • (edit) hbase-shell/src/main/ruby/shell/commands/remove_rsgroup.rb
          • (edit) hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsBase.java
          • (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/MasterObserver.java
          • (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseMasterAndRegionObserver.java
          • (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java
          • (edit) hbase-rsgroup/pom.xml
          • (edit) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminServer.java
          • (edit) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManagerImpl.java
          • (edit) hbase-shell/src/main/ruby/shell/commands/move_tables_rsgroup.rb
          • (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java
          • (edit) hbase-shell/src/main/ruby/shell/commands/get_server_rsgroup.rb
          • (edit) hbase-shell/src/main/ruby/shell/commands/get_rsgroup.rb
          • (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseMasterObserver.java
          • (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestWALObserver.java
          • (edit) hbase-shell/src/main/ruby/hbase/rsgroup_admin.rb
          • (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALFactory.java
          • (edit) hbase-shell/src/test/ruby/shell/rsgroup_shell_test.rb
          • (edit) hbase-shell/src/main/ruby/shell/commands/move_servers_rsgroup.rb
          • (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/ServerName.java
          • (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRollAbort.java
          • (edit) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupableBalancer.java
          • (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/wal/AbstractFSWALProvider.java
          • (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALReaderOnSecureWAL.java
          • (edit) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/Utility.java
          • (edit) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminEndpoint.java
          • (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/AbstractTestWALReplay.java
          • (edit) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminClient.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #2505 (See https://builds.apache.org/job/HBase-Trunk_matrix/2505/ ) HBASE-17624 Address late review of HBASE-6721 , rsgroups feature (stack: rev e019961150624236615fe7bd4a9017622d9f78d0) (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/LoadBalancer.java (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfo.java (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java (edit) hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroups.java (edit) hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/master/balancer/TestRSGroupBasedLoadBalancer.java (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java (edit) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupSerDe.java (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALSplit.java (delete) hbase-common/src/test/java/org/apache/hadoop/hbase/util/TestAddressing.java (edit) hbase-shell/src/main/ruby/shell/commands/get_table_rsgroup.rb (edit) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java (edit) hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestAsyncProcess.java (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestSplitLogManager.java (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java (edit) hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/VerifyingRSGroupAdminClient.java (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterCoprocessorHost.java (edit) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManager.java (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestDurability.java (edit) hbase-it/src/test/java/org/apache/hadoop/hbase/rsgroup/IntegrationTestRSGroup.java (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAdmin2.java (edit) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdmin.java (edit) hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsOfflineMode.java (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java (add) hbase-common/src/main/java/org/apache/hadoop/hbase/util/Address.java (edit) hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestSimpleRequestController.java (edit) hbase-shell/src/main/ruby/shell/commands/balance_rsgroup.rb (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/util/Addressing.java (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java (edit) hbase-shell/src/main/ruby/shell/commands/list_rsgroups.rb (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/TestServerName.java (edit) hbase-shell/src/main/ruby/shell/commands/remove_rsgroup.rb (edit) hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsBase.java (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/MasterObserver.java (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseMasterAndRegionObserver.java (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java (edit) hbase-rsgroup/pom.xml (edit) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminServer.java (edit) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManagerImpl.java (edit) hbase-shell/src/main/ruby/shell/commands/move_tables_rsgroup.rb (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java (edit) hbase-shell/src/main/ruby/shell/commands/get_server_rsgroup.rb (edit) hbase-shell/src/main/ruby/shell/commands/get_rsgroup.rb (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseMasterObserver.java (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestWALObserver.java (edit) hbase-shell/src/main/ruby/hbase/rsgroup_admin.rb (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALFactory.java (edit) hbase-shell/src/test/ruby/shell/rsgroup_shell_test.rb (edit) hbase-shell/src/main/ruby/shell/commands/move_servers_rsgroup.rb (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/ServerName.java (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRollAbort.java (edit) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupableBalancer.java (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/wal/AbstractFSWALProvider.java (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALReaderOnSecureWAL.java (edit) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/Utility.java (edit) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminEndpoint.java (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/AbstractTestWALReplay.java (edit) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminClient.java
          Hide
          busbey Sean Busbey added a comment -

          I understand the desire to keep this feature away from casual use by avoiding the ref guide, but I remain uncomfortable with the release note on a JIRA being the only docs on how to enable it, what it does, and what to worry about. Especially because that makes us less likely, as a community, to put in the needed hardening over time to open up for general use.

          I know I've joked before about writing a book on "The Undocumented HBase" to mixed responses, but what do folks think about starting docs outside of the ref guide for general "you might cut yourself" / "we aren't sure this is a good idea or staying" features so that we at least have a common starting point for invested in hbase looking to push what it can reasonably do with an experienced ops team?

          ...maybe I should post this to dev@? after the holidays?

          Show
          busbey Sean Busbey added a comment - I understand the desire to keep this feature away from casual use by avoiding the ref guide, but I remain uncomfortable with the release note on a JIRA being the only docs on how to enable it, what it does, and what to worry about. Especially because that makes us less likely, as a community, to put in the needed hardening over time to open up for general use. I know I've joked before about writing a book on "The Undocumented HBase" to mixed responses, but what do folks think about starting docs outside of the ref guide for general "you might cut yourself" / "we aren't sure this is a good idea or staying" features so that we at least have a common starting point for invested in hbase looking to push what it can reasonably do with an experienced ops team? ...maybe I should post this to dev@? after the holidays?
          Hide
          toffer Francis Liu added a comment -

          I'd say either of the two you'd prefer to start is good.

          Thanks Sean Busbey, I started the discuss thread. Also, I've moved the jiras.

          BTW, if there were a backport to branch-1 we'd probably try it. (Or if there were not a backport at the time, I might do it.)

          Thanks! BTW I think Enis Soztutar might already have a backport as he created one before.

          Show
          toffer Francis Liu added a comment - I'd say either of the two you'd prefer to start is good. Thanks Sean Busbey , I started the discuss thread. Also, I've moved the jiras. BTW, if there were a backport to branch-1 we'd probably try it. (Or if there were not a backport at the time, I might do it.) Thanks! BTW I think Enis Soztutar might already have a backport as he created one before.
          Hide
          apurtell Andrew Purtell added a comment -

          BTW, if there were a backport to branch-1 we'd probably try it. (Or if there were not a backport at the time, I might do it.)

          Show
          apurtell Andrew Purtell added a comment - BTW, if there were a backport to branch-1 we'd probably try it. (Or if there were not a backport at the time, I might do it.)
          Hide
          apurtell Andrew Purtell added a comment -

          As where I was on the MOB backport, I think it would be good to have evidence from functional, stability, and performance tests (the basics: PE, LTT, YCSB, ITs) that the feature does not impact folks who don't want it. The way group based assignment is implemented suggests we will get a good result.

          Show
          apurtell Andrew Purtell added a comment - As where I was on the MOB backport, I think it would be good to have evidence from functional, stability, and performance tests (the basics: PE, LTT, YCSB, ITs) that the feature does not impact folks who don't want it. The way group based assignment is implemented suggests we will get a good result.
          Hide
          busbey Sean Busbey added a comment -

          If this is going to go back earlier than 2.0, I would like a [DISCUSS] thread on what folks want to use as a gating threshold.

          I can start the thread. Is the discussion about region server groups specifically or about a more general gating threshold?

          In the past we've only done such threads for specific issues. I'd love to have a generally agreed upon set of gates for new features, but I'm not sure we'll reach a conclusion in a timely manner.

          I'd say either of the two you'd prefer to start is good.

          Show
          busbey Sean Busbey added a comment - If this is going to go back earlier than 2.0, I would like a [DISCUSS] thread on what folks want to use as a gating threshold. I can start the thread. Is the discussion about region server groups specifically or about a more general gating threshold? In the past we've only done such threads for specific issues. I'd love to have a generally agreed upon set of gates for new features, but I'm not sure we'll reach a conclusion in a timely manner. I'd say either of the two you'd prefer to start is good.
          Hide
          toffer Francis Liu added a comment -

          please move remaining open sub tasks to top-level tasks.

          Will do.

          If this is going to go back earlier than 2.0, I would like a [DISCUSS] thread on what folks want to use as a gating threshold.

          I can start the thread. Is the discussion about region server groups specifically or about a more general gating threshold?

          Show
          toffer Francis Liu added a comment - please move remaining open sub tasks to top-level tasks. Will do. If this is going to go back earlier than 2.0, I would like a [DISCUSS] thread on what folks want to use as a gating threshold. I can start the thread. Is the discussion about region server groups specifically or about a more general gating threshold?
          Hide
          busbey Sean Busbey added a comment -

          please move remaining open sub tasks to top-level tasks.

          If this is going to go back earlier than 2.0, I would like a [DISCUSS] thread on what folks want to use as a gating threshold.

          Show
          busbey Sean Busbey added a comment - please move remaining open sub tasks to top-level tasks. If this is going to go back earlier than 2.0, I would like a [DISCUSS] thread on what folks want to use as a gating threshold.
          Hide
          toffer Francis Liu added a comment -

          Awesome! Thanks a lot Enis Soztutar! Thanks a lot to all the reviewers and commenters!

          BTW given this is pluggable any chance we can backport this to 1.x and 0.98 as an experimental feature? I'd like to start using this version internally.

          Show
          toffer Francis Liu added a comment - Awesome! Thanks a lot Enis Soztutar ! Thanks a lot to all the reviewers and commenters! BTW given this is pluggable any chance we can backport this to 1.x and 0.98 as an experimental feature? I'd like to start using this version internally.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in HBase-Trunk_matrix #777 (See https://builds.apache.org/job/HBase-Trunk_matrix/777/)
          HBASE-6721 RegionServer Group based Assignment (Francis Liu) (enis: rev ca816f0780f6e5a117b85810cf35f3b29c964ddc)

          • hbase-shell/src/main/ruby/shell/commands/get_table_rsgroup.rb
          • hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/master/balancer/TestRSGroupBasedLoadBalancer.java
          • hbase-protocol/pom.xml
          • hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseMasterAndRegionObserver.java
          • hbase-shell/src/main/ruby/hbase/rsgroup_admin.rb
          • hbase-it/pom.xml
          • hbase-rsgroup/pom.xml
          • hbase-shell/src/main/ruby/shell/commands/balance_rsgroup.rb
          • hbase-shell/src/main/ruby/shell/commands/move_rsgroup_servers.rb
          • hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManager.java
          • hbase-shell/src/main/ruby/shell/commands/get_server_rsgroup.rb
          • hbase-server/src/main/java/org/apache/hadoop/hbase/master/LoadBalancer.java
          • hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
          • hbase-it/src/test/java/org/apache/hadoop/hbase/rsgroup/IntegrationTestRSGroup.java
          • hbase-shell/src/main/ruby/shell/commands/list_rsgroups.rb
          • hbase-shell/src/main/ruby/shell/commands/get_rsgroup.rb
          • hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/VerifyingRSGroupAdminClient.java
          • hbase-shell/src/test/rsgroup/org/apache/hadoop/hbase/client/rsgroup/TestShellRSGroups.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java
          • hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminServer.java
          • hbase-shell/src/test/java/org/apache/hadoop/hbase/client/TestShell.java
          • hbase-shell/src/main/ruby/shell/commands/add_rsgroup.rb
          • hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupableBalancer.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterCoprocessorHost.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/master/normalizer/TestSimpleRegionNormalizer.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterStatusServlet.java
          • hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroups.java
          • hbase-protocol/src/main/protobuf/RSGroupAdmin.proto
          • hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterServices.java
          • hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminEndpoint.java
          • hbase-common/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfo.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java
          • hbase-protocol/src/main/protobuf/RSGroup.proto
          • hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
          • hbase-shell/src/main/ruby/hbase.rb
          • hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RSGroupAdminProtos.java
          • hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java
          • hbase-shell/src/test/ruby/test_helper.rb
          • hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RSGroupProtos.java
          • hbase-shell/pom.xml
          • hbase-shell/src/test/ruby/shell/rsgroup_shell_test.rb
          • hbase-shell/src/main/ruby/shell/commands/move_rsgroup_tables.rb
          • hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/MasterObserver.java
          • hbase-shell/src/main/ruby/hbase/hbase.rb
          • hbase-shell/src/main/ruby/shell/commands/remove_rsgroup.rb
          • hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java
          • hbase-shell/src/main/ruby/shell.rb
          • hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseMasterObserver.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManagerOnCluster.java
          • hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupSerDe.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
          • pom.xml
          • hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsBase.java
          • hbase-shell/src/main/ruby/shell/commands.rb
          • hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManagerImpl.java
          • hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminClient.java
          • hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.jamon
          • hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdmin.java
          • hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsOfflineMode.java
          • hbase-common/src/main/java/org/apache/hadoop/hbase/ServerName.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in HBase-Trunk_matrix #777 (See https://builds.apache.org/job/HBase-Trunk_matrix/777/ ) HBASE-6721 RegionServer Group based Assignment (Francis Liu) (enis: rev ca816f0780f6e5a117b85810cf35f3b29c964ddc) hbase-shell/src/main/ruby/shell/commands/get_table_rsgroup.rb hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/master/balancer/TestRSGroupBasedLoadBalancer.java hbase-protocol/pom.xml hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseMasterAndRegionObserver.java hbase-shell/src/main/ruby/hbase/rsgroup_admin.rb hbase-it/pom.xml hbase-rsgroup/pom.xml hbase-shell/src/main/ruby/shell/commands/balance_rsgroup.rb hbase-shell/src/main/ruby/shell/commands/move_rsgroup_servers.rb hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManager.java hbase-shell/src/main/ruby/shell/commands/get_server_rsgroup.rb hbase-server/src/main/java/org/apache/hadoop/hbase/master/LoadBalancer.java hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java hbase-it/src/test/java/org/apache/hadoop/hbase/rsgroup/IntegrationTestRSGroup.java hbase-shell/src/main/ruby/shell/commands/list_rsgroups.rb hbase-shell/src/main/ruby/shell/commands/get_rsgroup.rb hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/VerifyingRSGroupAdminClient.java hbase-shell/src/test/rsgroup/org/apache/hadoop/hbase/client/rsgroup/TestShellRSGroups.java hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminServer.java hbase-shell/src/test/java/org/apache/hadoop/hbase/client/TestShell.java hbase-shell/src/main/ruby/shell/commands/add_rsgroup.rb hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupableBalancer.java hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterCoprocessorHost.java hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController.java hbase-server/src/test/java/org/apache/hadoop/hbase/master/normalizer/TestSimpleRegionNormalizer.java hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterStatusServlet.java hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroups.java hbase-protocol/src/main/protobuf/RSGroupAdmin.proto hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterServices.java hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminEndpoint.java hbase-common/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfo.java hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java hbase-protocol/src/main/protobuf/RSGroup.proto hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java hbase-shell/src/main/ruby/hbase.rb hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RSGroupAdminProtos.java hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java hbase-shell/src/test/ruby/test_helper.rb hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RSGroupProtos.java hbase-shell/pom.xml hbase-shell/src/test/ruby/shell/rsgroup_shell_test.rb hbase-shell/src/main/ruby/shell/commands/move_rsgroup_tables.rb hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/MasterObserver.java hbase-shell/src/main/ruby/hbase/hbase.rb hbase-shell/src/main/ruby/shell/commands/remove_rsgroup.rb hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java hbase-shell/src/main/ruby/shell.rb hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseMasterObserver.java hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManagerOnCluster.java hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupSerDe.java hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java pom.xml hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsBase.java hbase-shell/src/main/ruby/shell/commands.rb hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManagerImpl.java hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminClient.java hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.jamon hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdmin.java hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsOfflineMode.java hbase-common/src/main/java/org/apache/hadoop/hbase/ServerName.java
          Hide
          enis Enis Soztutar added a comment -

          Added a release note as well. Hopefully it is clear enough.

          Show
          enis Enis Soztutar added a comment - Added a release note as well. Hopefully it is clear enough.
          Hide
          enis Enis Soztutar added a comment -

          Committed this to master. Thanks Francis Liu and others.

          Show
          enis Enis Soztutar added a comment - Committed this to master. Thanks Francis Liu and others.
          Hide
          enis Enis Soztutar added a comment -

          Given the recent problems with hadoopqa, I've checked the https://builds.apache.org/job/PreCommit-HBASE-Build/896 manually which is a timed-out build. The findbugs and checkstyle issues are addressed. I've also run the unit tests on a node, and it passed except for 1 flaky test.

          I'll commit this shortly.

          Show
          enis Enis Soztutar added a comment - Given the recent problems with hadoopqa, I've checked the https://builds.apache.org/job/PreCommit-HBASE-Build/896 manually which is a timed-out build. The findbugs and checkstyle issues are addressed. I've also run the unit tests on a node, and it passed except for 1 flaky test. I'll commit this shortly.
          Hide
          toffer Francis Liu added a comment -

          Thanks Sean Busbey, please share a link to the build once it's running.

          Enis Soztutar Even tho the last run timed out, there are no longer any new checkstyle or findbugs issues.

          Show
          toffer Francis Liu added a comment - Thanks Sean Busbey , please share a link to the build once it's running. Enis Soztutar Even tho the last run timed out, there are no longer any new checkstyle or findbugs issues.
          Hide
          busbey Sean Busbey added a comment -

          thanks for finding an edge case for yetus.

          I'll spin up a clean build environment and see what yetus does there.

          Show
          busbey Sean Busbey added a comment - thanks for finding an edge case for yetus. I'll spin up a clean build environment and see what yetus does there.
          Hide
          toffer Francis Liu added a comment -

          Sean Busbey looks like the build timed out again but this time because each module was run twice?

          Show
          toffer Francis Liu added a comment - Sean Busbey looks like the build timed out again but this time because each module was run twice?
          Hide
          busbey Sean Busbey added a comment -

          Thanks for taking a look at this Sean Busbey. Sorry I meant "root", I mentioned component because the directory is called "component" which I realize now just makes things confusing. If this is a bug that'd be in Yetus?

          Ah. the directory is named component because of how I set up the jenkins job. That's just to distinguish it from "test_framework" in a way that I can copy/paste as I set up other projects.

          We're definitely running individual modules even after we've determined we have to run at the top level of the project. That's probably a bug in yetus, but I haven't confirmed yet that we don't do module reordering in our test personality. If we do, then the bug could be in our personality rather than yetus proper.

          Show
          busbey Sean Busbey added a comment - Thanks for taking a look at this Sean Busbey. Sorry I meant "root", I mentioned component because the directory is called "component" which I realize now just makes things confusing. If this is a bug that'd be in Yetus? Ah. the directory is named component because of how I set up the jenkins job. That's just to distinguish it from "test_framework" in a way that I can copy/paste as I set up other projects. We're definitely running individual modules even after we've determined we have to run at the top level of the project. That's probably a bug in yetus, but I haven't confirmed yet that we don't do module reordering in our test personality . If we do, then the bug could be in our personality rather than yetus proper.
          Hide
          toffer Francis Liu added a comment -

          Addressed findbugs and checkstyle comments.

          Findbugs warned about ServerName being serializable while HostAndPort is not. So I made HostAndPort transient and resolved lazily. Let me know if I should be addressing this in another way.

          Show
          toffer Francis Liu added a comment - Addressed findbugs and checkstyle comments. Findbugs warned about ServerName being serializable while HostAndPort is not. So I made HostAndPort transient and resolved lazily. Let me know if I should be addressing this in another way.
          Hide
          toffer Francis Liu added a comment -

          I don't see a test result for "hbase-component" in the yetus output. the hbase-server module is ridiculously slow and I see that we're running both at "root" (the top of the project) and other individual modules. That looks like a bug.

          Thanks for taking a look at this Sean Busbey. Sorry I meant "root", I mentioned component because the directory is called "component" which I realize now just makes things confusing. If this is a bug that'd be in Yetus?

          Show
          toffer Francis Liu added a comment - I don't see a test result for "hbase-component" in the yetus output. the hbase-server module is ridiculously slow and I see that we're running both at "root" (the top of the project) and other individual modules. That looks like a bug. Thanks for taking a look at this Sean Busbey . Sorry I meant "root", I mentioned component because the directory is called "component" which I realize now just makes things confusing. If this is a bug that'd be in Yetus?
          Hide
          busbey Sean Busbey added a comment -

          I started build #894 with the flags to tell yetus not to re-run unit tests per jdk.

          Show
          busbey Sean Busbey added a comment - I started build #894 with the flags to tell yetus not to re-run unit tests per jdk.
          Hide
          busbey Sean Busbey added a comment -

          Can someone rebase the patch so I can retry now that we've update to Yetus 0.2.0?

          And now I see that it is already doing this. Ignore this part

          Show
          busbey Sean Busbey added a comment - Can someone rebase the patch so I can retry now that we've update to Yetus 0.2.0? And now I see that it is already doing this. Ignore this part
          Hide
          busbey Sean Busbey added a comment -

          I don't see a test result for "hbase-component" in the yetus output. the hbase-server module is ridiculously slow and I see that we're running both at "root" (the top of the project) and other individual modules. That looks like a bug.

          Can someone rebase the patch so I can retry now that we've update to Yetus 0.2.0?

          There's an open discussion on the "Testing and CI" dev@hbase thread about switching to only doing unit tests with jdk7, which would also help keep us under the 500 minute limit by avoiding 2x hbase-server runs.

          Show
          busbey Sean Busbey added a comment - I don't see a test result for "hbase-component" in the yetus output. the hbase-server module is ridiculously slow and I see that we're running both at "root" (the top of the project) and other individual modules. That looks like a bug. Can someone rebase the patch so I can retry now that we've update to Yetus 0.2.0? There's an open discussion on the "Testing and CI" dev@hbase thread about switching to only doing unit tests with jdk7, which would also help keep us under the 500 minute limit by avoiding 2x hbase-server runs.
          Hide
          enis Enis Soztutar added a comment -

          The longest component unit test run by a large margin is hbase-server and hbase-component

          I am not sure what is /component. Maybe Sean Busbey can comment on it.

          At the worst we can run the tests locally (I can do that sometime this week). I saw a couple of checkstyle / findbugs issues. Did you have a chance to look into those?

          Show
          enis Enis Soztutar added a comment - The longest component unit test run by a large margin is hbase-server and hbase-component I am not sure what is /component. Maybe Sean Busbey can comment on it. At the worst we can run the tests locally (I can do that sometime this week). I saw a couple of checkstyle / findbugs issues. Did you have a chance to look into those?
          Hide
          toffer Francis Liu added a comment -

          Enis Soztutar Looks like the build aborted again due to 500mins timeout:

          https://builds.apache.org/job/PreCommit-HBASE-Build/843/console

          I'm wondering if it's really expected to finish in 500 mins?

          • Pre-unit tests (~160mins total elapsed)
          • jdk7 unit tests (410mins total elapsed)
            -jdk8....

          The longest component unit test run by a large margin is hbase-server and hbase-component. I compared the run times to other successful pre-commit builds and hbase-server runtime seems to be pretty similar. Tho I'm wondering why we are running hbase-server again when hbase-component has already been run? The redundancy seems to be what's making the build go over the time limit?

          Show
          toffer Francis Liu added a comment - Enis Soztutar Looks like the build aborted again due to 500mins timeout: https://builds.apache.org/job/PreCommit-HBASE-Build/843/console I'm wondering if it's really expected to finish in 500 mins? Pre-unit tests (~160mins total elapsed) jdk7 unit tests (410mins total elapsed) -jdk8.... The longest component unit test run by a large margin is hbase-server and hbase-component. I compared the run times to other successful pre-commit builds and hbase-server runtime seems to be pretty similar. Tho I'm wondering why we are running hbase-server again when hbase-component has already been run? The redundancy seems to be what's making the build go over the time limit?
          Hide
          toffer Francis Liu added a comment -

          rebased patch.

          Show
          toffer Francis Liu added a comment - rebased patch.
          Hide
          enis Enis Soztutar added a comment -

          rebase.

          Show
          enis Enis Soztutar added a comment - rebase.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          -1 patch 0m 4s HBASE-6721 does not apply to master. Rebase required? Wrong Branch? See https://yetus.apache.org/documentation/latest/precommit-patchnames for help.



          Subsystem Report/Notes
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12791388/hbase-6721-v27.patch
          JIRA Issue HBASE-6721
          Powered by Apache Yetus 0.1.0 http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HBASE-Build/828/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 patch 0m 4s HBASE-6721 does not apply to master. Rebase required? Wrong Branch? See https://yetus.apache.org/documentation/latest/precommit-patchnames for help. Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12791388/hbase-6721-v27.patch JIRA Issue HBASE-6721 Powered by Apache Yetus 0.1.0 http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HBASE-Build/828/console This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 patch 0m 2s The patch file was not named according to hbase's naming conventions. Please see https://yetus.apache.org/documentation/latest/precommit-patchnames for instructions.
          -1 patch 0m 6s HBASE-6721 does not apply to master. Rebase required? Wrong Branch? See https://yetus.apache.org/documentation/latest/precommit-patchnames for help.



          Subsystem Report/Notes
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12791376/hbase-6721-v27.patch.txt
          JIRA Issue HBASE-6721
          Powered by Apache Yetus 0.1.0 http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HBASE-Build/826/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 patch 0m 2s The patch file was not named according to hbase's naming conventions. Please see https://yetus.apache.org/documentation/latest/precommit-patchnames for instructions. -1 patch 0m 6s HBASE-6721 does not apply to master. Rebase required? Wrong Branch? See https://yetus.apache.org/documentation/latest/precommit-patchnames for help. Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12791376/hbase-6721-v27.patch.txt JIRA Issue HBASE-6721 Powered by Apache Yetus 0.1.0 http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HBASE-Build/826/console This message was automatically generated.
          Hide
          enis Enis Soztutar added a comment -

          Reattaching for hadoopqa.

          Show
          enis Enis Soztutar added a comment - Reattaching for hadoopqa.
          Hide
          toffer Francis Liu added a comment -

          Looks like the build aborted because it timed out.

          https://builds.apache.org/job/PreCommit-HBASE-Build/816/console

          Can we try and rerun the build? I'm going to try and run the tests on my local machine as well.

          Show
          toffer Francis Liu added a comment - Looks like the build aborted because it timed out. https://builds.apache.org/job/PreCommit-HBASE-Build/816/console Can we try and rerun the build? I'm going to try and run the tests on my local machine as well.
          Hide
          enis Enis Soztutar added a comment -

          It is active by default. If you don't specify the skip-rsgroup property the rsgroup module will get built.

          Ok, perfect. I did not see the <activeByDefault>true</activeByDefault> tag, so I was assuming the other way around.

          Not sure why the hadoopqa is not running for the latest patches.

          Show
          enis Enis Soztutar added a comment - It is active by default. If you don't specify the skip-rsgroup property the rsgroup module will get built. Ok, perfect. I did not see the <activeByDefault>true</activeByDefault> tag, so I was assuming the other way around. Not sure why the hadoopqa is not running for the latest patches.
          Hide
          toffer Francis Liu added a comment -

          If that is the case, it is still fine. If you don't want to activate, don't pass the property at all. Then how we can activate this with !skip-rsgroup ? I mean how do you activate the profile with the current patch without specifying -P?

          It is active by default. If you don't specify the skip-rsgroup property the rsgroup module will get built.

          ie

          1. builds hbase-rsgoup
            mvn install

          #does not build hbase-rsgroup
          mvn install -Dskip-rsgoup

          It's basically the same idea as -DskipITs and -DskipTests.

          Of course you can explicitly specify it via the -P switch as well:

          1. builds hbase-rsgoup
            mvn install -P rsgroup

          #does not build hbase-rsgroup
          mvn install -P !rsgroup

          Show
          toffer Francis Liu added a comment - If that is the case, it is still fine. If you don't want to activate, don't pass the property at all. Then how we can activate this with !skip-rsgroup ? I mean how do you activate the profile with the current patch without specifying -P? It is active by default. If you don't specify the skip-rsgroup property the rsgroup module will get built. ie builds hbase-rsgoup mvn install #does not build hbase-rsgroup mvn install -Dskip-rsgoup It's basically the same idea as -DskipITs and -DskipTests. Of course you can explicitly specify it via the -P switch as well: builds hbase-rsgoup mvn install -P rsgroup #does not build hbase-rsgroup mvn install -P !rsgroup
          Hide
          enis Enis Soztutar added a comment -

          So in your case 'mvn install -Drsgroup=false' would still enable the rsgroup profile?

          If that is the case, it is still fine. If you don't want to activate, don't pass the property at all. Then how we can activate this with !skip-rsgroup ? I mean how do you activate the profile with the current patch without specifying -P?

          Show
          enis Enis Soztutar added a comment - So in your case 'mvn install -Drsgroup=false' would still enable the rsgroup profile? If that is the case, it is still fine. If you don't want to activate, don't pass the property at all. Then how we can activate this with !skip-rsgroup ? I mean how do you activate the profile with the current patch without specifying -P ?
          Hide
          toffer Francis Liu added a comment -

          I forgot to rebase when I did rebase had a minor conflict. So I uploaded a rebased patch.

          Show
          toffer Francis Liu added a comment - I forgot to rebase when I did rebase had a minor conflict. So I uploaded a rebased patch.
          Hide
          toffer Francis Liu added a comment -

          Thanks Elliott Clark!

          Enis Soztutar doesn't the activation property actually only tests for the presence of a propery and ignores the value:

          http://maven.apache.org/guides/introduction/introduction-to-profiles.html

          So in your case 'mvn install -Drsgroup=false' would still enable the rsgroup profile?

          Show
          toffer Francis Liu added a comment - Thanks Elliott Clark ! Enis Soztutar doesn't the activation property actually only tests for the presence of a propery and ignores the value: http://maven.apache.org/guides/introduction/introduction-to-profiles.html So in your case 'mvn install -Drsgroup=false' would still enable the rsgroup profile?
          Hide
          enis Enis Soztutar added a comment -

          I skimmed the patch looking around the changes, looks good. One minor nit is that instead of mvn install -Dskip-rsgroup=false which is double negation, we can do mvn install -Drsgroup=true.

          Let's get a hadoopqa run. After we address the reported issues, I'll do the commit.

          Show
          enis Enis Soztutar added a comment - I skimmed the patch looking around the changes, looks good. One minor nit is that instead of mvn install -Dskip-rsgroup=false which is double negation, we can do mvn install -Drsgroup=true . Let's get a hadoopqa run. After we address the reported issues, I'll do the commit.
          Hide
          eclark Elliott Clark added a comment -

          Oh I was looking at the draft rather than the last version. I see that lots of stuff has been fixed. Thanks.

          Ok i'm officially -0.9999 this patch. Not minus one.

          Show
          eclark Elliott Clark added a comment - Oh I was looking at the draft rather than the last version. I see that lots of stuff has been fixed. Thanks. Ok i'm officially -0.9999 this patch. Not minus one.
          Hide
          toffer Francis Liu added a comment -

          So then the last part is getting better naming

          Are you referring to the rename from Group -> RSGroup? Then the just uploaded patch should've addressed it. Let me know if I missed anything. If not let me know what I'm missing?

          Show
          toffer Francis Liu added a comment - So then the last part is getting better naming Are you referring to the rename from Group -> RSGroup? Then the just uploaded patch should've addressed it. Let me know if I missed anything. If not let me know what I'm missing?
          Hide
          toffer Francis Liu added a comment -

          Attached patch which is pretty much done except for the negative port concern.

          Yeah, address the groups naming thing.

          Done. Change classes, packages, etc to RSGroups.

          Address the ServerName questions, breaking compat is not an option.

          It seems clear now the serverName form I fixed in TestMasterStatusServlet is indeed a typo. So the current patch can still support the old form. Tho it cannot support negative port numbers due to port restrictions. Let me know if this is not acceptable.

          Don't go around the Metrics to get to jmx

          Removed JMX this code from the base patch and filed HBASE-15382

          Don't create a new protobuf class when not needed

          Done

          Move it into a different module with a profile that doesn't include it.

          Done. Tho RSGroup protos are still in hbase-protocol and there's one POJO (RSGroupInfo) that had to stay in hbase-common as it is needed by the RSGroup related MasterObserver CP methods. RSGroup shell is also still in hbase-shell.

          Show
          toffer Francis Liu added a comment - Attached patch which is pretty much done except for the negative port concern. Yeah, address the groups naming thing. Done. Change classes, packages, etc to RSGroups. Address the ServerName questions, breaking compat is not an option. It seems clear now the serverName form I fixed in TestMasterStatusServlet is indeed a typo. So the current patch can still support the old form. Tho it cannot support negative port numbers due to port restrictions. Let me know if this is not acceptable. Don't go around the Metrics to get to jmx Removed JMX this code from the base patch and filed HBASE-15382 Don't create a new protobuf class when not needed Done Move it into a different module with a profile that doesn't include it. Done. Tho RSGroup protos are still in hbase-protocol and there's one POJO (RSGroupInfo) that had to stay in hbase-common as it is needed by the RSGroup related MasterObserver CP methods. RSGroup shell is also still in hbase-shell.
          Hide
          eclark Elliott Clark added a comment -

          The negative port numbers are unlikely to cause any errors so I would say we can ignore that. So then the last part is getting better naming for things.

          Show
          eclark Elliott Clark added a comment - The negative port numbers are unlikely to cause any errors so I would say we can ignore that. So then the last part is getting better naming for things.
          Hide
          toffer Francis Liu added a comment -

          HBASE-1502 is committed around 0.92 timeframe, so I am not sure whether we will have any such server name strings around anymore.

          Thanks for pointing that out Enis Soztutar. This is a clear indication that the ServerNameform in TestMasterStatusServlet is a typo 'rootserver:123,12345' as it is neither the old form or the new form. Then that leaves wether negative port numbers are a breaking change or not? As HostAndPort doesn't allow it.

          Show
          toffer Francis Liu added a comment - HBASE-1502 is committed around 0.92 timeframe, so I am not sure whether we will have any such server name strings around anymore. Thanks for pointing that out Enis Soztutar . This is a clear indication that the ServerNameform in TestMasterStatusServlet is a typo 'rootserver:123,12345' as it is neither the old form or the new form. Then that leaves wether negative port numbers are a breaking change or not? As HostAndPort doesn't allow it.
          Hide
          enis Enis Soztutar added a comment -

          Need to address circular dependecy between hbase-rsgroup and hbase-server. Since the MasterObserver CP has hooks into the GroupAdminEndpoint.

          If it is a strict "no" on the RS Group coprocessor reaching out to the internals of AccessController (as per above comment from Andrew Purtell), then the only option is to add the rs group specific methods to the base MasterObserver. I think it is acceptable since they would be empty method declarations if the module is not used.

          For the ServerName change, this parts seems relevant (from ServerName.parseFrom()):

              // The str returned could be old style -- pre hbase-1502 -- which was
              // hostname and port seperated by a colon rather than hostname, port and
              // startcode delimited by a ','.​
          

          HBASE-1502 is committed around 0.92 timeframe, so I am not sure whether we will have any such server name strings around anymore.

          Show
          enis Enis Soztutar added a comment - Need to address circular dependecy between hbase-rsgroup and hbase-server. Since the MasterObserver CP has hooks into the GroupAdminEndpoint. If it is a strict "no" on the RS Group coprocessor reaching out to the internals of AccessController (as per above comment from Andrew Purtell ), then the only option is to add the rs group specific methods to the base MasterObserver. I think it is acceptable since they would be empty method declarations if the module is not used. For the ServerName change, this parts seems relevant (from ServerName.parseFrom()): // The str returned could be old style -- pre hbase-1502 -- which was // hostname and port seperated by a colon rather than hostname, port and // startcode delimited by a ','.​ HBASE-1502 is committed around 0.92 timeframe, so I am not sure whether we will have any such server name strings around anymore.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 hbaseanti 0m 0s Patch does not have any anti-patterns.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 15 new or modified test files.
          +1 mvninstall 4m 5s master passed
          +1 compile 5m 35s master passed with JDK v1.8.0_72
          +1 compile 5m 5s master passed with JDK v1.7.0_95
          +1 checkstyle 25m 35s master passed
          +1 mvneclipse 3m 4s master passed
          +1 hbaseprotoc 4m 0s the patch passed
          -1 findbugs 10m 25s branch/. no findbugs output file (./target/findbugsXml.xml)
          +1 javadoc 3m 48s master passed with JDK v1.8.0_72
          +1 javadoc 4m 46s master passed with JDK v1.7.0_95
          +1 mvninstall 6m 30s the patch passed
          +1 compile 5m 18s the patch passed with JDK v1.8.0_72
          +1 cc 5m 18s the patch passed
          +1 javac 5m 18s the patch passed
          +1 compile 5m 32s the patch passed with JDK v1.7.0_95
          +1 cc 5m 32s the patch passed
          +1 javac 5m 32s the patch passed
          -1 checkstyle 13m 3s Patch generated 17 new checkstyle issues in root (total was 637, now 650).
          -1 checkstyle 1m 38s Patch generated 3 new checkstyle issues in hbase-client (total was 188, now 189).
          -1 checkstyle 0m 12s Patch generated 4 new checkstyle issues in hbase-rsgroup (total was 0, now 4).
          -1 checkstyle 3m 55s Patch generated 10 new checkstyle issues in hbase-server (total was 415, now 424).
          +1 mvneclipse 3m 16s the patch passed
          -1 rubocop 0m 28s The applied patch generated 137 new rubocop issues (total was 263, now 399).
          -1 ruby-lint 0m 11s The applied patch generated 68 new ruby-lint issues (total was 137, now 205).
          -1 whitespace 0m 1s The patch has 94 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          +1 hadoopcheck 29m 22s Patch does not cause any errors with Hadoop 2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1.
          +1 hbaseprotoc 3m 39s the patch passed
          -1 findbugs 10m 49s patch/. no findbugs output file (./target/findbugsXml.xml)
          -1 findbugs 1m 8s hbase-common introduced 3 new FindBugs issues.
          +1 javadoc 4m 0s the patch passed with JDK v1.8.0_72
          +1 javadoc 4m 55s the patch passed with JDK v1.7.0_95
          -1 unit 26m 7s root in the patch failed with JDK v1.8.0_72.
          +1 unit 1m 3s hbase-client in the patch passed with JDK v1.8.0_72.
          +1 unit 2m 1s hbase-common in the patch passed with JDK v1.8.0_72.
          +1 unit 0m 19s hbase-it in the patch passed with JDK v1.8.0_72.
          +1 unit 0m 29s hbase-protocol in the patch passed with JDK v1.8.0_72.
          +1 unit 0m 23s hbase-rsgroup in the patch passed with JDK v1.8.0_72.
          -1 unit 18m 19s hbase-server in the patch failed with JDK v1.8.0_72.
          -1 unit 7m 47s hbase-shell in the patch failed with JDK v1.8.0_72.
          -1 unit 27m 30s root in the patch failed with JDK v1.7.0_95.
          +1 unit 1m 3s hbase-client in the patch passed with JDK v1.7.0_95.
          +1 unit 1m 56s hbase-common in the patch passed with JDK v1.7.0_95.
          +1 unit 0m 20s hbase-it in the patch passed with JDK v1.7.0_95.
          +1 unit 0m 26s hbase-protocol in the patch passed with JDK v1.7.0_95.
          +1 unit 0m 23s hbase-rsgroup in the patch passed with JDK v1.7.0_95.
          -1 unit 18m 58s hbase-server in the patch failed with JDK v1.7.0_95.
          -1 unit 7m 16s hbase-shell in the patch failed with JDK v1.7.0_95.
          +1 asflicense 1m 26s Patch does not generate ASF License warnings.
          297m 3s



          Reason Tests
          FindBugs module:hbase-common
            Class org.apache.hadoop.hbase.ServerName defines non-transient non-serializable instance field hostAndPort In ServerName.java:instance field hostAndPort In ServerName.java
            Class org.apache.hadoop.hbase.rsgroup.GroupInfo defines non-transient non-serializable instance field servers In GroupInfo.java:instance field servers In GroupInfo.java
            Class org.apache.hadoop.hbase.rsgroup.GroupInfo defines non-transient non-serializable instance field tables In GroupInfo.java:instance field tables In GroupInfo.java
          JDK v1.8.0_72 Failed junit tests hadoop.hbase.ipc.TestRpcMetrics
            hadoop.hbase.regionserver.TestMetricsRegion
            hadoop.hbase.regionserver.TestMetricsRegionServer
            hadoop.hbase.client.TestShell
            hadoop.hbase.client.TestShellGroups
            hadoop.hbase.ipc.TestRpcMetrics
            hadoop.hbase.regionserver.TestMetricsRegion
            hadoop.hbase.regionserver.TestMetricsRegionServer
            hadoop.hbase.client.TestShell
            hadoop.hbase.client.TestShellGroups
          JDK v1.7.0_95 Failed junit tests hadoop.hbase.ipc.TestRpcMetrics
            hadoop.hbase.regionserver.TestMetricsRegion
            hadoop.hbase.regionserver.TestMetricsRegionServer
            hadoop.hbase.client.TestShell
            hadoop.hbase.client.TestShellGroups
            hadoop.hbase.ipc.TestRpcMetrics
            hadoop.hbase.regionserver.TestMetricsRegion
            hadoop.hbase.regionserver.TestMetricsRegionServer
            hadoop.hbase.client.TestShell
            hadoop.hbase.client.TestShellGroups



          Subsystem Report/Notes
          Docker Client=1.9.1 Server=1.9.1 Image:yetus/hbase:date2016-02-25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12789833/hbase-6721-v26_draft1.patch
          JIRA Issue HBASE-6721
          Optional Tests asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile xml cc hbaseprotoc rubocop ruby_lint
          uname Linux ae4c5edeceba 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
          git revision master / 6e9d355
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HBASE-Build/688/artifact/patchprocess/diff-checkstyle-root.txt
          checkstyle https://builds.apache.org/job/PreCommit-HBASE-Build/688/artifact/patchprocess/diff-checkstyle-hbase-client.txt
          checkstyle https://builds.apache.org/job/PreCommit-HBASE-Build/688/artifact/patchprocess/diff-checkstyle-hbase-rsgroup.txt
          checkstyle https://builds.apache.org/job/PreCommit-HBASE-Build/688/artifact/patchprocess/diff-checkstyle-hbase-server.txt
          rubocop v0.37.2
          rubocop https://builds.apache.org/job/PreCommit-HBASE-Build/688/artifact/patchprocess/diff-patch-rubocop.txt
          ruby-lint v2.1.0
          ruby-lint https://builds.apache.org/job/PreCommit-HBASE-Build/688/artifact/patchprocess/diff-patch-ruby-lint.txt
          whitespace https://builds.apache.org/job/PreCommit-HBASE-Build/688/artifact/patchprocess/whitespace-eol.txt
          findbugs https://builds.apache.org/job/PreCommit-HBASE-Build/688/artifact/patchprocess/new-findbugs-hbase-common.html
          unit https://builds.apache.org/job/PreCommit-HBASE-Build/688/artifact/patchprocess/patch-unit-root-jdk1.8.0_72.txt
          unit https://builds.apache.org/job/PreCommit-HBASE-Build/688/artifact/patchprocess/patch-unit-hbase-server-jdk1.8.0_72.txt
          unit https://builds.apache.org/job/PreCommit-HBASE-Build/688/artifact/patchprocess/patch-unit-hbase-shell-jdk1.8.0_72.txt
          unit https://builds.apache.org/job/PreCommit-HBASE-Build/688/artifact/patchprocess/patch-unit-root-jdk1.7.0_95.txt
          unit https://builds.apache.org/job/PreCommit-HBASE-Build/688/artifact/patchprocess/patch-unit-hbase-server-jdk1.7.0_95.txt
          unit https://builds.apache.org/job/PreCommit-HBASE-Build/688/artifact/patchprocess/patch-unit-hbase-shell-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HBASE-Build/688/artifact/patchprocess/patch-unit-root-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HBASE-Build/688/artifact/patchprocess/patch-unit-hbase-server-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HBASE-Build/688/artifact/patchprocess/patch-unit-hbase-shell-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HBASE-Build/688/artifact/patchprocess/patch-unit-root-jdk1.7.0_95.txt https://builds.apache.org/job/PreCommit-HBASE-Build/688/artifact/patchprocess/patch-unit-hbase-server-jdk1.7.0_95.txt https://builds.apache.org/job/PreCommit-HBASE-Build/688/artifact/patchprocess/patch-unit-hbase-shell-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HBASE-Build/688/testReport/
          modules C: . hbase-client hbase-common hbase-it hbase-protocol hbase-rsgroup hbase-server hbase-shell U: .
          Max memory used 195MB
          Powered by Apache Yetus 0.1.0 http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HBASE-Build/688/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 hbaseanti 0m 0s Patch does not have any anti-patterns. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 15 new or modified test files. +1 mvninstall 4m 5s master passed +1 compile 5m 35s master passed with JDK v1.8.0_72 +1 compile 5m 5s master passed with JDK v1.7.0_95 +1 checkstyle 25m 35s master passed +1 mvneclipse 3m 4s master passed +1 hbaseprotoc 4m 0s the patch passed -1 findbugs 10m 25s branch/. no findbugs output file (./target/findbugsXml.xml) +1 javadoc 3m 48s master passed with JDK v1.8.0_72 +1 javadoc 4m 46s master passed with JDK v1.7.0_95 +1 mvninstall 6m 30s the patch passed +1 compile 5m 18s the patch passed with JDK v1.8.0_72 +1 cc 5m 18s the patch passed +1 javac 5m 18s the patch passed +1 compile 5m 32s the patch passed with JDK v1.7.0_95 +1 cc 5m 32s the patch passed +1 javac 5m 32s the patch passed -1 checkstyle 13m 3s Patch generated 17 new checkstyle issues in root (total was 637, now 650). -1 checkstyle 1m 38s Patch generated 3 new checkstyle issues in hbase-client (total was 188, now 189). -1 checkstyle 0m 12s Patch generated 4 new checkstyle issues in hbase-rsgroup (total was 0, now 4). -1 checkstyle 3m 55s Patch generated 10 new checkstyle issues in hbase-server (total was 415, now 424). +1 mvneclipse 3m 16s the patch passed -1 rubocop 0m 28s The applied patch generated 137 new rubocop issues (total was 263, now 399). -1 ruby-lint 0m 11s The applied patch generated 68 new ruby-lint issues (total was 137, now 205). -1 whitespace 0m 1s The patch has 94 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 xml 0m 2s The patch has no ill-formed XML file. +1 hadoopcheck 29m 22s Patch does not cause any errors with Hadoop 2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. +1 hbaseprotoc 3m 39s the patch passed -1 findbugs 10m 49s patch/. no findbugs output file (./target/findbugsXml.xml) -1 findbugs 1m 8s hbase-common introduced 3 new FindBugs issues. +1 javadoc 4m 0s the patch passed with JDK v1.8.0_72 +1 javadoc 4m 55s the patch passed with JDK v1.7.0_95 -1 unit 26m 7s root in the patch failed with JDK v1.8.0_72. +1 unit 1m 3s hbase-client in the patch passed with JDK v1.8.0_72. +1 unit 2m 1s hbase-common in the patch passed with JDK v1.8.0_72. +1 unit 0m 19s hbase-it in the patch passed with JDK v1.8.0_72. +1 unit 0m 29s hbase-protocol in the patch passed with JDK v1.8.0_72. +1 unit 0m 23s hbase-rsgroup in the patch passed with JDK v1.8.0_72. -1 unit 18m 19s hbase-server in the patch failed with JDK v1.8.0_72. -1 unit 7m 47s hbase-shell in the patch failed with JDK v1.8.0_72. -1 unit 27m 30s root in the patch failed with JDK v1.7.0_95. +1 unit 1m 3s hbase-client in the patch passed with JDK v1.7.0_95. +1 unit 1m 56s hbase-common in the patch passed with JDK v1.7.0_95. +1 unit 0m 20s hbase-it in the patch passed with JDK v1.7.0_95. +1 unit 0m 26s hbase-protocol in the patch passed with JDK v1.7.0_95. +1 unit 0m 23s hbase-rsgroup in the patch passed with JDK v1.7.0_95. -1 unit 18m 58s hbase-server in the patch failed with JDK v1.7.0_95. -1 unit 7m 16s hbase-shell in the patch failed with JDK v1.7.0_95. +1 asflicense 1m 26s Patch does not generate ASF License warnings. 297m 3s Reason Tests FindBugs module:hbase-common   Class org.apache.hadoop.hbase.ServerName defines non-transient non-serializable instance field hostAndPort In ServerName.java:instance field hostAndPort In ServerName.java   Class org.apache.hadoop.hbase.rsgroup.GroupInfo defines non-transient non-serializable instance field servers In GroupInfo.java:instance field servers In GroupInfo.java   Class org.apache.hadoop.hbase.rsgroup.GroupInfo defines non-transient non-serializable instance field tables In GroupInfo.java:instance field tables In GroupInfo.java JDK v1.8.0_72 Failed junit tests hadoop.hbase.ipc.TestRpcMetrics   hadoop.hbase.regionserver.TestMetricsRegion   hadoop.hbase.regionserver.TestMetricsRegionServer   hadoop.hbase.client.TestShell   hadoop.hbase.client.TestShellGroups   hadoop.hbase.ipc.TestRpcMetrics   hadoop.hbase.regionserver.TestMetricsRegion   hadoop.hbase.regionserver.TestMetricsRegionServer   hadoop.hbase.client.TestShell   hadoop.hbase.client.TestShellGroups JDK v1.7.0_95 Failed junit tests hadoop.hbase.ipc.TestRpcMetrics   hadoop.hbase.regionserver.TestMetricsRegion   hadoop.hbase.regionserver.TestMetricsRegionServer   hadoop.hbase.client.TestShell   hadoop.hbase.client.TestShellGroups   hadoop.hbase.ipc.TestRpcMetrics   hadoop.hbase.regionserver.TestMetricsRegion   hadoop.hbase.regionserver.TestMetricsRegionServer   hadoop.hbase.client.TestShell   hadoop.hbase.client.TestShellGroups Subsystem Report/Notes Docker Client=1.9.1 Server=1.9.1 Image:yetus/hbase:date2016-02-25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12789833/hbase-6721-v26_draft1.patch JIRA Issue HBASE-6721 Optional Tests asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile xml cc hbaseprotoc rubocop ruby_lint uname Linux ae4c5edeceba 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh git revision master / 6e9d355 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HBASE-Build/688/artifact/patchprocess/diff-checkstyle-root.txt checkstyle https://builds.apache.org/job/PreCommit-HBASE-Build/688/artifact/patchprocess/diff-checkstyle-hbase-client.txt checkstyle https://builds.apache.org/job/PreCommit-HBASE-Build/688/artifact/patchprocess/diff-checkstyle-hbase-rsgroup.txt checkstyle https://builds.apache.org/job/PreCommit-HBASE-Build/688/artifact/patchprocess/diff-checkstyle-hbase-server.txt rubocop v0.37.2 rubocop https://builds.apache.org/job/PreCommit-HBASE-Build/688/artifact/patchprocess/diff-patch-rubocop.txt ruby-lint v2.1.0 ruby-lint https://builds.apache.org/job/PreCommit-HBASE-Build/688/artifact/patchprocess/diff-patch-ruby-lint.txt whitespace https://builds.apache.org/job/PreCommit-HBASE-Build/688/artifact/patchprocess/whitespace-eol.txt findbugs https://builds.apache.org/job/PreCommit-HBASE-Build/688/artifact/patchprocess/new-findbugs-hbase-common.html unit https://builds.apache.org/job/PreCommit-HBASE-Build/688/artifact/patchprocess/patch-unit-root-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-HBASE-Build/688/artifact/patchprocess/patch-unit-hbase-server-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-HBASE-Build/688/artifact/patchprocess/patch-unit-hbase-shell-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-HBASE-Build/688/artifact/patchprocess/patch-unit-root-jdk1.7.0_95.txt unit https://builds.apache.org/job/PreCommit-HBASE-Build/688/artifact/patchprocess/patch-unit-hbase-server-jdk1.7.0_95.txt unit https://builds.apache.org/job/PreCommit-HBASE-Build/688/artifact/patchprocess/patch-unit-hbase-shell-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HBASE-Build/688/artifact/patchprocess/patch-unit-root-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HBASE-Build/688/artifact/patchprocess/patch-unit-hbase-server-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HBASE-Build/688/artifact/patchprocess/patch-unit-hbase-shell-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HBASE-Build/688/artifact/patchprocess/patch-unit-root-jdk1.7.0_95.txt https://builds.apache.org/job/PreCommit-HBASE-Build/688/artifact/patchprocess/patch-unit-hbase-server-jdk1.7.0_95.txt https://builds.apache.org/job/PreCommit-HBASE-Build/688/artifact/patchprocess/patch-unit-hbase-shell-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HBASE-Build/688/testReport/ modules C: . hbase-client hbase-common hbase-it hbase-protocol hbase-rsgroup hbase-server hbase-shell U: . Max memory used 195MB Powered by Apache Yetus 0.1.0 http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HBASE-Build/688/console This message was automatically generated.
          Hide
          toffer Francis Liu added a comment -

          Attached WIP rebased patch which addresses the following:

          Yeah, address the groups naming thing.

          Partially done. I renamed the main proto classes and package name to Group -> RSGroup. Should I just blindly prepend all entities with Group in it to RSGroup?

          Address the ServerName questions, breaking compat is not an option.

          Waiting on response from Elliott Clark

          Don't go around the Metrics to get to jmx

          Waiting on response from Elliott Clark

          Don't create a new protobuf class when not needed

          Done

          Move it into a different module with a profile that doesn't include it.

          Partially done. Most of the classes are in a different module. Need to address circular dependecy between hbase-rsgroup and hbase-server. Since the MasterObserver CP has hooks into the GroupAdminEndpoint. Pom file needs a bit of streamlining as well.

          Show
          toffer Francis Liu added a comment - Attached WIP rebased patch which addresses the following: Yeah, address the groups naming thing. Partially done. I renamed the main proto classes and package name to Group -> RSGroup. Should I just blindly prepend all entities with Group in it to RSGroup? Address the ServerName questions, breaking compat is not an option. Waiting on response from Elliott Clark Don't go around the Metrics to get to jmx Waiting on response from Elliott Clark Don't create a new protobuf class when not needed Done Move it into a different module with a profile that doesn't include it. Partially done. Most of the classes are in a different module. Need to address circular dependecy between hbase-rsgroup and hbase-server. Since the MasterObserver CP has hooks into the GroupAdminEndpoint. Pom file needs a bit of streamlining as well.
          Hide
          toffer Francis Liu added a comment -

          Yeah hadoop can do whatever they want. I would prefer that HBase is consistent whenever interacting with jmx so when we make changes in the future it's all moving from the same base.

          Took a look at this further it may be possible to reuse tags to publish metadata in some form but I would need to add APIs to the base (BaseSource?). Tho IMHO we could probably do better by providing a common base on which to publish metadata via jmx instead of retrofitting it into hadoop-metrics which looking at the direction hadoop went is not how they intend metrics subsystem to be used. So instead of addressing this in this patch I propose we create two new jiras: 1. create apis to publish metadata via jmx, 2. create jira to leverage #1 and publish region server groups metadata. Thoughts?

          Both formats have been able to be parsed for a long time. So if the test needs to be changed to pass then that's breaking change. If not then we shouldn't have changes that aren't needed in a patch this large.

          I'm not sure how the format used by TestMasterStatusServlet: rootserver:123,12345 is actually a valid format. It seems like a typo/bug to me. For a few reasons:

          1. The format is not documented in the javadoc
          2. If you look at how the format mentioned is parsed the components would come out as host=rootserver:123/port=12345/startcode=12345. Which doesn't make much sense? Also ':' is not a valid character for a hostname according to RFC.

          What uses this format? Let me know what I'm missing. On a related note if we support this format we really can't use HostAndPort as it assumes ':' will delimit a port and will throw an exception if it finds it in the hostname.

          On a related note another behavior change since the introduction of HostAndPort is that '-1' can no longer be used as a port number (or any negative number). Would this be a breaking change? Let me know how you'd like to go about this change.

          Show
          toffer Francis Liu added a comment - Yeah hadoop can do whatever they want. I would prefer that HBase is consistent whenever interacting with jmx so when we make changes in the future it's all moving from the same base. Took a look at this further it may be possible to reuse tags to publish metadata in some form but I would need to add APIs to the base (BaseSource?). Tho IMHO we could probably do better by providing a common base on which to publish metadata via jmx instead of retrofitting it into hadoop-metrics which looking at the direction hadoop went is not how they intend metrics subsystem to be used. So instead of addressing this in this patch I propose we create two new jiras: 1. create apis to publish metadata via jmx, 2. create jira to leverage #1 and publish region server groups metadata. Thoughts? Both formats have been able to be parsed for a long time. So if the test needs to be changed to pass then that's breaking change. If not then we shouldn't have changes that aren't needed in a patch this large. I'm not sure how the format used by TestMasterStatusServlet: rootserver:123,12345 is actually a valid format. It seems like a typo/bug to me. For a few reasons: 1. The format is not documented in the javadoc 2. If you look at how the format mentioned is parsed the components would come out as host=rootserver:123/port=12345/startcode=12345. Which doesn't make much sense? Also ':' is not a valid character for a hostname according to RFC. What uses this format? Let me know what I'm missing. On a related note if we support this format we really can't use HostAndPort as it assumes ':' will delimit a port and will throw an exception if it finds it in the hostname. On a related note another behavior change since the introduction of HostAndPort is that '-1' can no longer be used as a port number (or any negative number). Would this be a breaking change? Let me know how you'd like to go about this change.
          Hide
          eclark Elliott Clark added a comment -

          Took a quick look at how the NN publishes metadata via JMX.

          Yeah hadoop can do whatever they want. I would prefer that HBase is consistent whenever interacting with jmx so when we make changes in the future it's all moving from the same base.

          I think you misread the change.

          Both formats have been able to be parsed for a long time. So if the test needs to be changed to pass then that's breaking change. If not then we shouldn't have changes that aren't needed in a patch this large.

          Show
          eclark Elliott Clark added a comment - Took a quick look at how the NN publishes metadata via JMX. Yeah hadoop can do whatever they want. I would prefer that HBase is consistent whenever interacting with jmx so when we make changes in the future it's all moving from the same base. I think you misread the change. Both formats have been able to be parsed for a long time. So if the test needs to be changed to pass then that's breaking change. If not then we shouldn't have changes that aren't needed in a patch this large.
          Hide
          toffer Francis Liu added a comment -

          Took a quick look at how the NN publishes metadata via JMX. And for the HA information at least it does it the same way this patch is doing in for HBase:

          https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java#L1727

          Just in case it wasn't common knowledge. I'll publish the information in whatever form is acceptable.

          Show
          toffer Francis Liu added a comment - Took a quick look at how the NN publishes metadata via JMX. And for the HA information at least it does it the same way this patch is doing in for HBase: https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java#L1727 Just in case it wasn't common knowledge. I'll publish the information in whatever form is acceptable.
          Hide
          toffer Francis Liu added a comment -

          Thanks for the list Elliott Clark. Some comments.

          > Yeah, address the groups naming thing.

          Sounds good.

          > moving in its own module seems easy since most of the code is already isolated, but there are few things that we never done, like having the shell commands in a module different than hbase-shell

          The previous agreement was to keep the gorup commands in hbase-shell the same as security before. Would guys still be ok with this? Elliott Clark Matteo Bertozzi

          > Address the ServerName questions, breaking compat is not an option.

          I think you misread the change. The code change was to actually fix a broken unit test as it was using colon as a delimiter to serverName. I just fixed it by changing the colon to ','. If this is not the case please feel free to point it out in RB so I can address it properly.

          > Don't go around the Metrics to get to jmx

          The reason I went around metrics was because the information being published is not really what is consumed as metrics, it's basically just group information (name, tables, servers, etc) . It'd be ugly but probably possible to shoehorn this into the metrics subsystem. Is this really how we do this even for non-metrics related information to be published? There's alot less flexibility in how we can structure the metadata.

          > Don't create a new protobuf class when not needed

          This was actually a review comment made by Jonathan Hsieh. I'll take a look if ServerName can suffice as a substitute.

          > Move it into a different module with a profile that doesn't include it.

          Will do.

          > If/when those are addressed then I would remove my -1. I would be a -0.9 but I wouldn't block it.

          Thanks.

          Show
          toffer Francis Liu added a comment - Thanks for the list Elliott Clark . Some comments. > Yeah, address the groups naming thing. Sounds good. > moving in its own module seems easy since most of the code is already isolated, but there are few things that we never done, like having the shell commands in a module different than hbase-shell The previous agreement was to keep the gorup commands in hbase-shell the same as security before. Would guys still be ok with this? Elliott Clark Matteo Bertozzi > Address the ServerName questions, breaking compat is not an option. I think you misread the change. The code change was to actually fix a broken unit test as it was using colon as a delimiter to serverName. I just fixed it by changing the colon to ','. If this is not the case please feel free to point it out in RB so I can address it properly. > Don't go around the Metrics to get to jmx The reason I went around metrics was because the information being published is not really what is consumed as metrics, it's basically just group information (name, tables, servers, etc) . It'd be ugly but probably possible to shoehorn this into the metrics subsystem. Is this really how we do this even for non-metrics related information to be published? There's alot less flexibility in how we can structure the metadata. > Don't create a new protobuf class when not needed This was actually a review comment made by Jonathan Hsieh . I'll take a look if ServerName can suffice as a substitute. > Move it into a different module with a profile that doesn't include it. Will do. > If/when those are addressed then I would remove my -1. I would be a -0.9 but I wouldn't block it. Thanks.
          Hide
          stack stack added a comment -

          Thanks Elliott Clark

          How about we commit what we have here to a branch and then work on the list above to get it committed? I'm game for helping out on fixing the ServerName stuff, the protobuf, and moving the thing to its own module (shell-work, etc.).

          Show
          stack stack added a comment - Thanks Elliott Clark How about we commit what we have here to a branch and then work on the list above to get it committed? I'm game for helping out on fixing the ServerName stuff, the protobuf, and moving the thing to its own module (shell-work, etc.).
          Hide
          eclark Elliott Clark added a comment -

          That is up for the users to decide.

          And up the the PMC to steer people in the correct direction based upon having deep knowledge of running HBase in production. So I'm doing that.

          Elliott Clark if was in a separate module and feature name was changed to not be 'group', would you your -1?

          Yeah, address the groups naming thing.
          Address the ServerName questions, breaking compat is not an option.
          Don't go around the Metrics to get to jmx
          Don't create a new protobuf class when not needed
          Move it into a different module with a profile that doesn't include it.

          If/when those are addressed then I would remove my -1. I would be a -0.9 but I wouldn't block it.

          Show
          eclark Elliott Clark added a comment - That is up for the users to decide. And up the the PMC to steer people in the correct direction based upon having deep knowledge of running HBase in production. So I'm doing that. Elliott Clark if was in a separate module and feature name was changed to not be 'group', would you your -1? Yeah, address the groups naming thing. Address the ServerName questions, breaking compat is not an option. Don't go around the Metrics to get to jmx Don't create a new protobuf class when not needed Move it into a different module with a profile that doesn't include it. If/when those are addressed then I would remove my -1. I would be a -0.9 but I wouldn't block it.
          Hide
          stack stack added a comment -

          Elliott Clark if was in a separate module and feature name was changed to not be 'group', would you your -1? Seems like it well isolated currently going by Matteo review above. Could hack on shell so it picked up commands at runtime so we could have downstream modules add commands....

          Show
          stack stack added a comment - Elliott Clark if was in a separate module and feature name was changed to not be 'group', would you your -1? Seems like it well isolated currently going by Matteo review above. Could hack on shell so it picked up commands at runtime so we could have downstream modules add commands....
          Hide
          mbertozzi Matteo Bertozzi added a comment -

          just gave a look at the patch. most of the code seems isolated.
          the client is separate from Admin with its own GroupAdmin
          the rpc is not in the master but in its own GroupAdminEndpoint
          the "group assignment" logic seems isolated in GroupBasedLoadBalancer
          and there are not many changes to the core.

          The only "big" thing that should be fixed are that too generic Group name and maybe doing something around that processBogusAssignments()/BOGUS_SERVER_NAME

          moving in its own module seems easy since most of the code is already isolated, but there are few things that we never done, like having the shell commands in a module different than hbase-shell

          Show
          mbertozzi Matteo Bertozzi added a comment - just gave a look at the patch. most of the code seems isolated. the client is separate from Admin with its own GroupAdmin the rpc is not in the master but in its own GroupAdminEndpoint the "group assignment" logic seems isolated in GroupBasedLoadBalancer and there are not many changes to the core. The only "big" thing that should be fixed are that too generic Group name and maybe doing something around that processBogusAssignments()/BOGUS_SERVER_NAME moving in its own module seems easy since most of the code is already isolated, but there are few things that we never done, like having the shell commands in a module different than hbase-shell
          Hide
          stack stack added a comment -

          What we going to do here?

          One of our biggest users has been working on getting this patch landed for years now. They put up a design and tried to address commentary as it came in. The patch has gone through many iterations. It even got deployed by another committer on a cluster and exercised. Implementation could be better but our internals are a little ahem, messy, so hard to overlay a new facility. A case has been made that this feature makes hbase palatable at scale, at least for how it is being used at Y!

          On the other hand, Elliott Clark objections are wholesome/valid. By default, this new code should not run. Having it out in a module doesn't sound like a bad idea if optional on and a bunch of new code. Yes, overloading 'group' will confuse as we progress. You are probably busy Elliott Clark but any chance of a list of what it would take to get you to drop your -1.

          Thanks

          Show
          stack stack added a comment - What we going to do here? One of our biggest users has been working on getting this patch landed for years now. They put up a design and tried to address commentary as it came in. The patch has gone through many iterations. It even got deployed by another committer on a cluster and exercised. Implementation could be better but our internals are a little ahem, messy, so hard to overlay a new facility. A case has been made that this feature makes hbase palatable at scale, at least for how it is being used at Y! On the other hand, Elliott Clark objections are wholesome/valid. By default, this new code should not run. Having it out in a module doesn't sound like a bad idea if optional on and a bunch of new code. Yes, overloading 'group' will confuse as we progress. You are probably busy Elliott Clark but any chance of a list of what it would take to get you to drop your -1. Thanks
          Hide
          clayb Clay B. added a comment -

          I would like to register that I am not at Yahoo! but as an operator of a large number of clusters will certainly take advantage of this feature. In fact, Guorui's work was motivated by the need for this feature here.

          Show
          clayb Clay B. added a comment - I would like to register that I am not at Yahoo! but as an operator of a large number of clusters will certainly take advantage of this feature. In fact, Guorui's work was motivated by the need for this feature here.
          Hide
          enis Enis Soztutar added a comment -

          The exact same was requested for memcached block cache, and it was reasonable then. I'm simply asking for this feature to get the same treatment that optional off by default removable features should get

          Yep, everybody agrees that it should be completely optional and non-core for this. What I am saying is that there is no need to fork out a different module for this. I just checked, nobody asked the memcache thing to be a different module above.

          This feature should never be used by anyone other than yahoo and we have a duty to our users to make sure that they understand that.

          That is up for the users to decide. Sophisticated users can make their own decisions.

          rsgroup as used in the table name is better.

          Agreed. I've brought this up in the review already. I thought we have addressed the cases, but if we haven't, we should stick with rsgroups rather than groups. Having consistent naming is something we are lacking in hbase (alter table in shell vs modify table in java, etc).

          Show
          enis Enis Soztutar added a comment - The exact same was requested for memcached block cache, and it was reasonable then. I'm simply asking for this feature to get the same treatment that optional off by default removable features should get Yep, everybody agrees that it should be completely optional and non-core for this. What I am saying is that there is no need to fork out a different module for this. I just checked, nobody asked the memcache thing to be a different module above. This feature should never be used by anyone other than yahoo and we have a duty to our users to make sure that they understand that. That is up for the users to decide. Sophisticated users can make their own decisions. rsgroup as used in the table name is better. Agreed. I've brought this up in the review already. I thought we have addressed the cases, but if we haven't, we should stick with rsgroups rather than groups. Having consistent naming is something we are lacking in hbase (alter table in shell vs modify table in java, etc).
          Hide
          eclark Elliott Clark added a comment -

          I think the current patch covers the already-agreed-upon requirement that the feature should be non-core and optional.

          The exact same was requested for memcached block cache, and it was reasonable then. I'm simply asking for this feature to get the same treatment that optional off by default removable features should get. This feature should never be used by anyone other than yahoo and we have a duty to our users to make sure that they understand that.

          On top of that the naming on this is awful. Groups are something that users are a part of. Its a term that been around unix for a very long time. The term is used all over the Hadoop and HBase code. We shouldn't overload the term. rsgroup as used in the table name is better.
          Why all the changes to comma for server name rather than colon ? Isn't that a breaking change? If it's not required then we should add tests, not change them.
          Going around the current metrics system is a code smell.
          Start code is optional on ServerName so no need to create a new protobuf.

          I have not withdrawn my explicit -1 as per the voting rules of Apache.

          Show
          eclark Elliott Clark added a comment - I think the current patch covers the already-agreed-upon requirement that the feature should be non-core and optional. The exact same was requested for memcached block cache, and it was reasonable then. I'm simply asking for this feature to get the same treatment that optional off by default removable features should get. This feature should never be used by anyone other than yahoo and we have a duty to our users to make sure that they understand that. On top of that the naming on this is awful. Groups are something that users are a part of. Its a term that been around unix for a very long time. The term is used all over the Hadoop and HBase code. We shouldn't overload the term. rsgroup as used in the table name is better. Why all the changes to comma for server name rather than colon ? Isn't that a breaking change? If it's not required then we should add tests, not change them. Going around the current metrics system is a code smell. Start code is optional on ServerName so no need to create a new protobuf. I have not withdrawn my explicit -1 as per the voting rules of Apache.
          Hide
          enis Enis Soztutar added a comment -

          There is a +1 outstanding for this patch. To reiterate the above comment, I think the current patch covers the already-agreed-upon requirement that the feature should be non-core and optional. We have not talked about needing a different module previously, and I am not sure why it is necessary. Unless, I hear otherwise by Friday, I'll go ahead with committing this.

          Show
          enis Enis Soztutar added a comment - There is a +1 outstanding for this patch. To reiterate the above comment, I think the current patch covers the already-agreed-upon requirement that the feature should be non-core and optional. We have not talked about needing a different module previously, and I am not sure why it is necessary. Unless, I hear otherwise by Friday, I'll go ahead with committing this.
          Hide
          devaraj Devaraj Das added a comment -

          Elliott Clark ping. Please respond when you have a moment.

          Show
          devaraj Devaraj Das added a comment - Elliott Clark ping. Please respond when you have a moment.
          Hide
          enis Enis Soztutar added a comment -

          With this patch, the RS groups is completely optional and non-core. I don't think having the code in a different module helps in any way.
          The patch corresponds to the plan that Francis put up here: https://issues.apache.org/jira/browse/HBASE-6721?focusedCommentId=14728516&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14728516. I thought there was an agreement already for this. We did not talk about having a different module in that.

          Show
          enis Enis Soztutar added a comment - With this patch, the RS groups is completely optional and non-core. I don't think having the code in a different module helps in any way. The patch corresponds to the plan that Francis put up here: https://issues.apache.org/jira/browse/HBASE-6721?focusedCommentId=14728516&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14728516 . I thought there was an agreement already for this. We did not talk about having a different module in that.
          Hide
          eclark Elliott Clark added a comment - - edited

          From what I can see most of my comments still stand. It's still built into the default client. It's still in the main module. Though I do appreciate that it's a co-processor.

          If we can move the co-processor to a different module and move the methods off the main admin classes I would be fine with it.

          Show
          eclark Elliott Clark added a comment - - edited From what I can see most of my comments still stand. It's still built into the default client. It's still in the main module. Though I do appreciate that it's a co-processor. If we can move the co-processor to a different module and move the methods off the main admin classes I would be fine with it.
          Hide
          yuzhihong@gmail.com Ted Yu added a comment -

          Any more review comments ?

          Show
          yuzhihong@gmail.com Ted Yu added a comment - Any more review comments ?
          Hide
          yuzhihong@gmail.com Ted Yu added a comment -

          +1

          Francis:
          Please update Release Note.

          Show
          yuzhihong@gmail.com Ted Yu added a comment - +1 Francis: Please update Release Note.
          Hide
          yuzhihong@gmail.com Ted Yu added a comment -
          hbase-server/target/generated-jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.java	UnusedImportsCheck	20	21
          hbase-server/target/generated-jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmplImpl.java	IndentationCheck	61	62
          

          To my knowledge, HBASE-15015 (committed after the QA run) should have disabled the above check.

          Show
          yuzhihong@gmail.com Ted Yu added a comment - hbase-server/target/generated-jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.java UnusedImportsCheck 20 21 hbase-server/target/generated-jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmplImpl.java IndentationCheck 61 62 To my knowledge, HBASE-15015 (committed after the QA run) should have disabled the above check.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12778630/hbase-6721-v25.patch
          against master branch at commit cdca22a36d0e2ec87cea732e0769c4fa4fa37e57.
          ATTACHMENT ID: 12778630

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 48 new or modified tests.

          +1 hadoop versions. The patch compiles with all supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0 2.7.1)

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 protoc. The applied patch does not increase the total number of protoc compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 checkstyle. The applied patch generated new checkstyle errors. Check build console for list of new errors.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 lineLengths. The patch introduces the following lines longer than 100:
          + * <code>rpc GetGroupInfo(.hbase.pb.GetGroupInfoRequest) returns (.hbase.pb.GetGroupInfoResponse);</code>
          + * <code>rpc GetGroupInfoOfTable(.hbase.pb.GetGroupInfoOfTableRequest) returns (.hbase.pb.GetGroupInfoOfTableResponse);</code>
          + * <code>rpc GetGroupInfoOfServer(.hbase.pb.GetGroupInfoOfServerRequest) returns (.hbase.pb.GetGroupInfoOfServerResponse);</code>
          + * <code>rpc MoveServers(.hbase.pb.MoveServersRequest) returns (.hbase.pb.MoveServersResponse);</code>
          + * <code>rpc MoveTables(.hbase.pb.MoveTablesRequest) returns (.hbase.pb.MoveTablesResponse);</code>
          + * <code>rpc RemoveGroup(.hbase.pb.RemoveGroupRequest) returns (.hbase.pb.RemoveGroupResponse);</code>
          + * <code>rpc BalanceGroup(.hbase.pb.BalanceGroupRequest) returns (.hbase.pb.BalanceGroupResponse);</code>
          + * <code>rpc ListGroupInfos(.hbase.pb.ListGroupInfosRequest) returns (.hbase.pb.ListGroupInfosResponse);</code>
          + * <code>rpc GetGroupInfo(.hbase.pb.GetGroupInfoRequest) returns (.hbase.pb.GetGroupInfoResponse);</code>
          + * <code>rpc GetGroupInfoOfTable(.hbase.pb.GetGroupInfoOfTableRequest) returns (.hbase.pb.GetGroupInfoOfTableResponse);</code>

          +1 site. The mvn post-site goal succeeds with this patch.

          +1 core tests. The patch passed unit tests in .

          +1 zombies. No zombie tests found running at the end of the build.

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/16934//testReport/
          Release Findbugs (version 2.0.3) warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/16934//artifact/patchprocess/newFindbugsWarnings.html
          Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/16934//artifact/patchprocess/checkstyle-aggregate.html

          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/16934//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12778630/hbase-6721-v25.patch against master branch at commit cdca22a36d0e2ec87cea732e0769c4fa4fa37e57. ATTACHMENT ID: 12778630 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 48 new or modified tests. +1 hadoop versions . The patch compiles with all supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0 2.7.1) +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 protoc . The applied patch does not increase the total number of protoc compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. -1 checkstyle . The applied patch generated new checkstyle errors. Check build console for list of new errors. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 lineLengths . The patch introduces the following lines longer than 100: + * <code>rpc GetGroupInfo(.hbase.pb.GetGroupInfoRequest) returns (.hbase.pb.GetGroupInfoResponse);</code> + * <code>rpc GetGroupInfoOfTable(.hbase.pb.GetGroupInfoOfTableRequest) returns (.hbase.pb.GetGroupInfoOfTableResponse);</code> + * <code>rpc GetGroupInfoOfServer(.hbase.pb.GetGroupInfoOfServerRequest) returns (.hbase.pb.GetGroupInfoOfServerResponse);</code> + * <code>rpc MoveServers(.hbase.pb.MoveServersRequest) returns (.hbase.pb.MoveServersResponse);</code> + * <code>rpc MoveTables(.hbase.pb.MoveTablesRequest) returns (.hbase.pb.MoveTablesResponse);</code> + * <code>rpc RemoveGroup(.hbase.pb.RemoveGroupRequest) returns (.hbase.pb.RemoveGroupResponse);</code> + * <code>rpc BalanceGroup(.hbase.pb.BalanceGroupRequest) returns (.hbase.pb.BalanceGroupResponse);</code> + * <code>rpc ListGroupInfos(.hbase.pb.ListGroupInfosRequest) returns (.hbase.pb.ListGroupInfosResponse);</code> + * <code>rpc GetGroupInfo(.hbase.pb.GetGroupInfoRequest) returns (.hbase.pb.GetGroupInfoResponse);</code> + * <code>rpc GetGroupInfoOfTable(.hbase.pb.GetGroupInfoOfTableRequest) returns (.hbase.pb.GetGroupInfoOfTableResponse);</code> +1 site . The mvn post-site goal succeeds with this patch. +1 core tests . The patch passed unit tests in . +1 zombies . No zombie tests found running at the end of the build. Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/16934//testReport/ Release Findbugs (version 2.0.3) warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/16934//artifact/patchprocess/newFindbugsWarnings.html Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/16934//artifact/patchprocess/checkstyle-aggregate.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/16934//console This message is automatically generated.
          Hide
          toffer Francis Liu added a comment -

          rebased patch

          Show
          toffer Francis Liu added a comment - rebased patch
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12778573/hbase-6721-v23.patch
          against master branch at commit eb59d4d7690bb4d8d0afa460202c68885ef6a271.
          ATTACHMENT ID: 12778573

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 48 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/16931//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12778573/hbase-6721-v23.patch against master branch at commit eb59d4d7690bb4d8d0afa460202c68885ef6a271. ATTACHMENT ID: 12778573 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 48 new or modified tests. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/16931//console This message is automatically generated.
          Hide
          toffer Francis Liu added a comment -

          Looks like I can only influence the import order via the jamon file (which was a mess to begin with). The rest cannot be fixed it seems.

          Show
          toffer Francis Liu added a comment - Looks like I can only influence the import order via the jamon file (which was a mess to begin with). The rest cannot be fixed it seems.
          Hide
          toffer Francis Liu added a comment -

          The remaining checkstyle failures are mainly because of the auto-generated MasterStatus java files. Do we need to fix this?

          hbase-server/target/generated-jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmplImpl.java	IndentationCheck	61	62
          hbase-server/target/generated-jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmplImpl.java	ImportOrderCheck	12	13
          hbase-server/target/generated-jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.java	ImportOrderCheck	12	13
          hbase-server/target/generated-jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.java	UnusedImportsCheck	20	21
          
          Show
          toffer Francis Liu added a comment - The remaining checkstyle failures are mainly because of the auto-generated MasterStatus java files. Do we need to fix this? hbase-server/target/generated-jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmplImpl.java IndentationCheck 61 62 hbase-server/target/generated-jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmplImpl.java ImportOrderCheck 12 13 hbase-server/target/generated-jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.java ImportOrderCheck 12 13 hbase-server/target/generated-jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.java UnusedImportsCheck 20 21
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12778306/hbase-6721-v22.patch
          against master branch at commit 80fc18d2e1f72d312a4ab90a32d6e44c9b560493.
          ATTACHMENT ID: 12778306

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 48 new or modified tests.

          +1 hadoop versions. The patch compiles with all supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0 2.7.1)

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 protoc. The applied patch does not increase the total number of protoc compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 checkstyle. The applied patch generated new checkstyle errors. Check build console for list of new errors.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 lineLengths. The patch introduces the following lines longer than 100:
          + * <code>rpc GetGroupInfo(.hbase.pb.GetGroupInfoRequest) returns (.hbase.pb.GetGroupInfoResponse);</code>
          + * <code>rpc GetGroupInfoOfTable(.hbase.pb.GetGroupInfoOfTableRequest) returns (.hbase.pb.GetGroupInfoOfTableResponse);</code>
          + * <code>rpc GetGroupInfoOfServer(.hbase.pb.GetGroupInfoOfServerRequest) returns (.hbase.pb.GetGroupInfoOfServerResponse);</code>
          + * <code>rpc MoveServers(.hbase.pb.MoveServersRequest) returns (.hbase.pb.MoveServersResponse);</code>
          + * <code>rpc MoveTables(.hbase.pb.MoveTablesRequest) returns (.hbase.pb.MoveTablesResponse);</code>
          + * <code>rpc RemoveGroup(.hbase.pb.RemoveGroupRequest) returns (.hbase.pb.RemoveGroupResponse);</code>
          + * <code>rpc BalanceGroup(.hbase.pb.BalanceGroupRequest) returns (.hbase.pb.BalanceGroupResponse);</code>
          + * <code>rpc ListGroupInfos(.hbase.pb.ListGroupInfosRequest) returns (.hbase.pb.ListGroupInfosResponse);</code>
          + * <code>rpc GetGroupInfo(.hbase.pb.GetGroupInfoRequest) returns (.hbase.pb.GetGroupInfoResponse);</code>
          + * <code>rpc GetGroupInfoOfTable(.hbase.pb.GetGroupInfoOfTableRequest) returns (.hbase.pb.GetGroupInfoOfTableResponse);</code>

          +1 site. The mvn post-site goal succeeds with this patch.

          +1 core tests. The patch passed unit tests in .

          +1 zombies. No zombie tests found running at the end of the build.

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/16912//testReport/
          Release Findbugs (version 2.0.3) warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/16912//artifact/patchprocess/newFindbugsWarnings.html
          Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/16912//artifact/patchprocess/checkstyle-aggregate.html

          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/16912//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12778306/hbase-6721-v22.patch against master branch at commit 80fc18d2e1f72d312a4ab90a32d6e44c9b560493. ATTACHMENT ID: 12778306 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 48 new or modified tests. +1 hadoop versions . The patch compiles with all supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0 2.7.1) +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 protoc . The applied patch does not increase the total number of protoc compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. -1 checkstyle . The applied patch generated new checkstyle errors. Check build console for list of new errors. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 lineLengths . The patch introduces the following lines longer than 100: + * <code>rpc GetGroupInfo(.hbase.pb.GetGroupInfoRequest) returns (.hbase.pb.GetGroupInfoResponse);</code> + * <code>rpc GetGroupInfoOfTable(.hbase.pb.GetGroupInfoOfTableRequest) returns (.hbase.pb.GetGroupInfoOfTableResponse);</code> + * <code>rpc GetGroupInfoOfServer(.hbase.pb.GetGroupInfoOfServerRequest) returns (.hbase.pb.GetGroupInfoOfServerResponse);</code> + * <code>rpc MoveServers(.hbase.pb.MoveServersRequest) returns (.hbase.pb.MoveServersResponse);</code> + * <code>rpc MoveTables(.hbase.pb.MoveTablesRequest) returns (.hbase.pb.MoveTablesResponse);</code> + * <code>rpc RemoveGroup(.hbase.pb.RemoveGroupRequest) returns (.hbase.pb.RemoveGroupResponse);</code> + * <code>rpc BalanceGroup(.hbase.pb.BalanceGroupRequest) returns (.hbase.pb.BalanceGroupResponse);</code> + * <code>rpc ListGroupInfos(.hbase.pb.ListGroupInfosRequest) returns (.hbase.pb.ListGroupInfosResponse);</code> + * <code>rpc GetGroupInfo(.hbase.pb.GetGroupInfoRequest) returns (.hbase.pb.GetGroupInfoResponse);</code> + * <code>rpc GetGroupInfoOfTable(.hbase.pb.GetGroupInfoOfTableRequest) returns (.hbase.pb.GetGroupInfoOfTableResponse);</code> +1 site . The mvn post-site goal succeeds with this patch. +1 core tests . The patch passed unit tests in . +1 zombies . No zombie tests found running at the end of the build. Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/16912//testReport/ Release Findbugs (version 2.0.3) warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/16912//artifact/patchprocess/newFindbugsWarnings.html Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/16912//artifact/patchprocess/checkstyle-aggregate.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/16912//console This message is automatically generated.
          Hide
          enis Enis Soztutar added a comment -

          Javadoc is not related. I think there is an issue tracking that already.

          Show
          enis Enis Soztutar added a comment - Javadoc is not related. I think there is an issue tracking that already.
          Hide
          toffer Francis Liu added a comment -

          Sorry missed that. Fixed in updated patch.

          Show
          toffer Francis Liu added a comment - Sorry missed that. Fixed in updated patch.
          Hide
          yuzhihong@gmail.com Ted Yu added a comment -

          There're some import order check warnings left:

          ./dev-support/checkstyle_report.py trunkCheckstyle.xml patchCheckstyle.xml
          hbase-client/src/main/java/org/apache/hadoop/hbase/group/GroupAdminClient.java ImportOrderCheck 0 1
          hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java RedundantImportCheck 0 4
          hbase-server/target/generated-jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.java ImportOrderCheck 12 13
          hbase-server/target/generated-jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.java UnusedImportsCheck 20 21
          hbase-server/target/generated-jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmplImpl.java IndentationCheck 61 62
          hbase-server/target/generated-jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmplImpl.java ImportOrderCheck 12 13

          Show
          yuzhihong@gmail.com Ted Yu added a comment - There're some import order check warnings left: ./dev-support/checkstyle_report.py trunkCheckstyle.xml patchCheckstyle.xml hbase-client/src/main/java/org/apache/hadoop/hbase/group/GroupAdminClient.java ImportOrderCheck 0 1 hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java RedundantImportCheck 0 4 hbase-server/target/generated-jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.java ImportOrderCheck 12 13 hbase-server/target/generated-jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.java UnusedImportsCheck 20 21 hbase-server/target/generated-jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmplImpl.java IndentationCheck 61 62 hbase-server/target/generated-jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmplImpl.java ImportOrderCheck 12 13
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12778218/hbase-6721-v21.patch
          against master branch at commit d78eddfdc8bad5068600e28a039276cc55063ce2.
          ATTACHMENT ID: 12778218

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 48 new or modified tests.

          +1 hadoop versions. The patch compiles with all supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0 2.7.1)

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 protoc. The applied patch does not increase the total number of protoc compiler warnings.

          -1 javadoc. The javadoc tool appears to have generated 1 warning messages.

          -1 checkstyle. The applied patch generated new checkstyle errors. Check build console for list of new errors.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 lineLengths. The patch introduces the following lines longer than 100:
          + * <code>rpc GetGroupInfo(.hbase.pb.GetGroupInfoRequest) returns (.hbase.pb.GetGroupInfoResponse);</code>
          + * <code>rpc GetGroupInfoOfTable(.hbase.pb.GetGroupInfoOfTableRequest) returns (.hbase.pb.GetGroupInfoOfTableResponse);</code>
          + * <code>rpc GetGroupInfoOfServer(.hbase.pb.GetGroupInfoOfServerRequest) returns (.hbase.pb.GetGroupInfoOfServerResponse);</code>
          + * <code>rpc MoveServers(.hbase.pb.MoveServersRequest) returns (.hbase.pb.MoveServersResponse);</code>
          + * <code>rpc MoveTables(.hbase.pb.MoveTablesRequest) returns (.hbase.pb.MoveTablesResponse);</code>
          + * <code>rpc RemoveGroup(.hbase.pb.RemoveGroupRequest) returns (.hbase.pb.RemoveGroupResponse);</code>
          + * <code>rpc BalanceGroup(.hbase.pb.BalanceGroupRequest) returns (.hbase.pb.BalanceGroupResponse);</code>
          + * <code>rpc ListGroupInfos(.hbase.pb.ListGroupInfosRequest) returns (.hbase.pb.ListGroupInfosResponse);</code>
          + * <code>rpc GetGroupInfo(.hbase.pb.GetGroupInfoRequest) returns (.hbase.pb.GetGroupInfoResponse);</code>
          + * <code>rpc GetGroupInfoOfTable(.hbase.pb.GetGroupInfoOfTableRequest) returns (.hbase.pb.GetGroupInfoOfTableResponse);</code>

          +1 site. The mvn post-site goal succeeds with this patch.

          +1 core tests. The patch passed unit tests in .

          +1 zombies. No zombie tests found running at the end of the build.

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/16903//testReport/
          Release Findbugs (version 2.0.3) warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/16903//artifact/patchprocess/newFindbugsWarnings.html
          Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/16903//artifact/patchprocess/checkstyle-aggregate.html

          Javadoc warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/16903//artifact/patchprocess/patchJavadocWarnings.txt
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/16903//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12778218/hbase-6721-v21.patch against master branch at commit d78eddfdc8bad5068600e28a039276cc55063ce2. ATTACHMENT ID: 12778218 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 48 new or modified tests. +1 hadoop versions . The patch compiles with all supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0 2.7.1) +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 protoc . The applied patch does not increase the total number of protoc compiler warnings. -1 javadoc . The javadoc tool appears to have generated 1 warning messages. -1 checkstyle . The applied patch generated new checkstyle errors. Check build console for list of new errors. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 lineLengths . The patch introduces the following lines longer than 100: + * <code>rpc GetGroupInfo(.hbase.pb.GetGroupInfoRequest) returns (.hbase.pb.GetGroupInfoResponse);</code> + * <code>rpc GetGroupInfoOfTable(.hbase.pb.GetGroupInfoOfTableRequest) returns (.hbase.pb.GetGroupInfoOfTableResponse);</code> + * <code>rpc GetGroupInfoOfServer(.hbase.pb.GetGroupInfoOfServerRequest) returns (.hbase.pb.GetGroupInfoOfServerResponse);</code> + * <code>rpc MoveServers(.hbase.pb.MoveServersRequest) returns (.hbase.pb.MoveServersResponse);</code> + * <code>rpc MoveTables(.hbase.pb.MoveTablesRequest) returns (.hbase.pb.MoveTablesResponse);</code> + * <code>rpc RemoveGroup(.hbase.pb.RemoveGroupRequest) returns (.hbase.pb.RemoveGroupResponse);</code> + * <code>rpc BalanceGroup(.hbase.pb.BalanceGroupRequest) returns (.hbase.pb.BalanceGroupResponse);</code> + * <code>rpc ListGroupInfos(.hbase.pb.ListGroupInfosRequest) returns (.hbase.pb.ListGroupInfosResponse);</code> + * <code>rpc GetGroupInfo(.hbase.pb.GetGroupInfoRequest) returns (.hbase.pb.GetGroupInfoResponse);</code> + * <code>rpc GetGroupInfoOfTable(.hbase.pb.GetGroupInfoOfTableRequest) returns (.hbase.pb.GetGroupInfoOfTableResponse);</code> +1 site . The mvn post-site goal succeeds with this patch. +1 core tests . The patch passed unit tests in . +1 zombies . No zombie tests found running at the end of the build. Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/16903//testReport/ Release Findbugs (version 2.0.3) warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/16903//artifact/patchprocess/newFindbugsWarnings.html Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/16903//artifact/patchprocess/checkstyle-aggregate.html Javadoc warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/16903//artifact/patchprocess/patchJavadocWarnings.txt Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/16903//console This message is automatically generated.
          Hide
          toffer Francis Liu added a comment -

          addressed checkstyle issues.

          Show
          toffer Francis Liu added a comment - addressed checkstyle issues.
          Hide
          toffer Francis Liu added a comment -

          I upload v20 on RB. Will address the checkstyle.

          Show
          toffer Francis Liu added a comment - I upload v20 on RB. Will address the checkstyle.
          Hide
          yuzhihong@gmail.com Ted Yu added a comment -

          I see.
          TestShellGroups passes locally.

          Mind addressing checkstyle warnings ?

          Patch on review board is v18.
          Please upload v20 (or v21) there.

          Thanks

          Show
          yuzhihong@gmail.com Ted Yu added a comment - I see. TestShellGroups passes locally. Mind addressing checkstyle warnings ? Patch on review board is v18. Please upload v20 (or v21) there. Thanks
          Hide
          toffer Francis Liu added a comment -

          I pretty much did that with TestShellGroups class, I just used the 'shell.test' property to define what test to be run intead of shell.test.include/shell.test.exclude. The reason the TestShell was failing was because you copied the TestShell class and did not exclude group_shell_test.rb similar to how replication_admin_test.rb is set. So we will need to update TestShell once it's restored so it doesn't fail because it tries to run group_shell_test.rb.

          Show
          toffer Francis Liu added a comment - I pretty much did that with TestShellGroups class, I just used the 'shell.test' property to define what test to be run intead of shell.test.include/shell.test.exclude. The reason the TestShell was failing was because you copied the TestShell class and did not exclude group_shell_test.rb similar to how replication_admin_test.rb is set. So we will need to update TestShell once it's restored so it doesn't fail because it tries to run group_shell_test.rb.
          Hide
          yuzhihong@gmail.com Ted Yu added a comment -

          include the group rb file in the TestShell class

          TestShell works by excluding replication related shell tests:

            public void testRunShellTests() throws IOException {
              System.setProperty("shell.test.exclude", "replication_admin_test.rb");
          

          Can you refer to hbase-shell//src/test/java/org/apache/hadoop/hbase/client/TestReplicationShell.java in branch-1.1 and create TestRegionServerGroupShell.java ?
          This way, we don't need to wait for restoration of TestShell.

          Show
          yuzhihong@gmail.com Ted Yu added a comment - include the group rb file in the TestShell class TestShell works by excluding replication related shell tests: public void testRunShellTests() throws IOException { System .setProperty( "shell.test.exclude" , "replication_admin_test.rb" ); Can you refer to hbase-shell//src/test/java/org/apache/hadoop/hbase/client/TestReplicationShell.java in branch-1.1 and create TestRegionServerGroupShell.java ? This way, we don't need to wait for restoration of TestShell.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12777930/hbase-6721-v20.patch
          against master branch at commit 60d33ce34191533bb858852584bd9bddfeb16a23.
          ATTACHMENT ID: 12777930

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 48 new or modified tests.

          +1 hadoop versions. The patch compiles with all supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0 2.7.1)

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 protoc. The applied patch does not increase the total number of protoc compiler warnings.

          -1 javadoc. The javadoc tool appears to have generated 1 warning messages.

          -1 checkstyle. The applied patch generated new checkstyle errors. Check build console for list of new errors.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 lineLengths. The patch introduces the following lines longer than 100:
          + * <code>rpc GetGroupInfo(.hbase.pb.GetGroupInfoRequest) returns (.hbase.pb.GetGroupInfoResponse);</code>
          + * <code>rpc GetGroupInfoOfTable(.hbase.pb.GetGroupInfoOfTableRequest) returns (.hbase.pb.GetGroupInfoOfTableResponse);</code>
          + * <code>rpc GetGroupInfoOfServer(.hbase.pb.GetGroupInfoOfServerRequest) returns (.hbase.pb.GetGroupInfoOfServerResponse);</code>
          + * <code>rpc MoveServers(.hbase.pb.MoveServersRequest) returns (.hbase.pb.MoveServersResponse);</code>
          + * <code>rpc MoveTables(.hbase.pb.MoveTablesRequest) returns (.hbase.pb.MoveTablesResponse);</code>
          + * <code>rpc RemoveGroup(.hbase.pb.RemoveGroupRequest) returns (.hbase.pb.RemoveGroupResponse);</code>
          + * <code>rpc BalanceGroup(.hbase.pb.BalanceGroupRequest) returns (.hbase.pb.BalanceGroupResponse);</code>
          + * <code>rpc ListGroupInfos(.hbase.pb.ListGroupInfosRequest) returns (.hbase.pb.ListGroupInfosResponse);</code>
          + * <code>rpc GetGroupInfo(.hbase.pb.GetGroupInfoRequest) returns (.hbase.pb.GetGroupInfoResponse);</code>
          + * <code>rpc GetGroupInfoOfTable(.hbase.pb.GetGroupInfoOfTableRequest) returns (.hbase.pb.GetGroupInfoOfTableResponse);</code>

          +1 site. The mvn post-site goal succeeds with this patch.

          +1 core tests. The patch passed unit tests in .

          +1 zombies. No zombie tests found running at the end of the build.

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/16873//testReport/
          Release Findbugs (version 2.0.3) warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/16873//artifact/patchprocess/newFindbugsWarnings.html
          Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/16873//artifact/patchprocess/checkstyle-aggregate.html

          Javadoc warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/16873//artifact/patchprocess/patchJavadocWarnings.txt
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/16873//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12777930/hbase-6721-v20.patch against master branch at commit 60d33ce34191533bb858852584bd9bddfeb16a23. ATTACHMENT ID: 12777930 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 48 new or modified tests. +1 hadoop versions . The patch compiles with all supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0 2.7.1) +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 protoc . The applied patch does not increase the total number of protoc compiler warnings. -1 javadoc . The javadoc tool appears to have generated 1 warning messages. -1 checkstyle . The applied patch generated new checkstyle errors. Check build console for list of new errors. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 lineLengths . The patch introduces the following lines longer than 100: + * <code>rpc GetGroupInfo(.hbase.pb.GetGroupInfoRequest) returns (.hbase.pb.GetGroupInfoResponse);</code> + * <code>rpc GetGroupInfoOfTable(.hbase.pb.GetGroupInfoOfTableRequest) returns (.hbase.pb.GetGroupInfoOfTableResponse);</code> + * <code>rpc GetGroupInfoOfServer(.hbase.pb.GetGroupInfoOfServerRequest) returns (.hbase.pb.GetGroupInfoOfServerResponse);</code> + * <code>rpc MoveServers(.hbase.pb.MoveServersRequest) returns (.hbase.pb.MoveServersResponse);</code> + * <code>rpc MoveTables(.hbase.pb.MoveTablesRequest) returns (.hbase.pb.MoveTablesResponse);</code> + * <code>rpc RemoveGroup(.hbase.pb.RemoveGroupRequest) returns (.hbase.pb.RemoveGroupResponse);</code> + * <code>rpc BalanceGroup(.hbase.pb.BalanceGroupRequest) returns (.hbase.pb.BalanceGroupResponse);</code> + * <code>rpc ListGroupInfos(.hbase.pb.ListGroupInfosRequest) returns (.hbase.pb.ListGroupInfosResponse);</code> + * <code>rpc GetGroupInfo(.hbase.pb.GetGroupInfoRequest) returns (.hbase.pb.GetGroupInfoResponse);</code> + * <code>rpc GetGroupInfoOfTable(.hbase.pb.GetGroupInfoOfTableRequest) returns (.hbase.pb.GetGroupInfoOfTableResponse);</code> +1 site . The mvn post-site goal succeeds with this patch. +1 core tests . The patch passed unit tests in . +1 zombies . No zombie tests found running at the end of the build. Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/16873//testReport/ Release Findbugs (version 2.0.3) warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/16873//artifact/patchprocess/newFindbugsWarnings.html Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/16873//artifact/patchprocess/checkstyle-aggregate.html Javadoc warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/16873//artifact/patchprocess/patchJavadocWarnings.txt Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/16873//console This message is automatically generated.
          Hide
          toffer Francis Liu added a comment -

          Looks like TestShell has been temporarily removed as part of test stabilization experiment in HBASE-14678.

          Given how it looks like when it was still there:

          https://github.com/apache/hbase/commit/f670649f0e2cfba8a2112eb5c1d79b8b7f52c3b2

          All I need to do is include the group rb file in the TestShell class. So we can either apply the TestShell change once it's been re-enabled. Or commit the group shell unit tests as a follow-up once TestShell has been re-enabled. Thoughts?

          Show
          toffer Francis Liu added a comment - Looks like TestShell has been temporarily removed as part of test stabilization experiment in HBASE-14678 . Given how it looks like when it was still there: https://github.com/apache/hbase/commit/f670649f0e2cfba8a2112eb5c1d79b8b7f52c3b2 All I need to do is include the group rb file in the TestShell class. So we can either apply the TestShell change once it's been re-enabled. Or commit the group shell unit tests as a follow-up once TestShell has been re-enabled. Thoughts?
          Hide
          toffer Francis Liu added a comment -

          addressed failing non-shell unit tests

          Show
          toffer Francis Liu added a comment - addressed failing non-shell unit tests
          Hide
          toffer Francis Liu added a comment -

          Took a quick look at GroupShellTest which misses master coprocessor setup similar to the following:

          Precisely, I created a separate ShellTest subclass as I needed to configure a different balancer and coprocessor which would not work with the existing shell unit tests. I think there's a bigger problem, that there is no proper way to run shell unit tests in trunk. I've filed HBASE-14981 to address that. I'm going to address the non-shell unit test failures for now.

          Show
          toffer Francis Liu added a comment - Took a quick look at GroupShellTest which misses master coprocessor setup similar to the following: Precisely, I created a separate ShellTest subclass as I needed to configure a different balancer and coprocessor which would not work with the existing shell unit tests. I think there's a bigger problem, that there is no proper way to run shell unit tests in trunk. I've filed HBASE-14981 to address that. I'm going to address the non-shell unit test failures for now.
          Hide
          yuzhihong@gmail.com Ted Yu added a comment -

          Took a quick look at GroupShellTest which misses master coprocessor setup similar to the following:

          +    TEST_UTIL.getConfiguration().set(CoprocessorHost.MASTER_COPROCESSOR_CONF_KEY,
          +        GroupAdminEndpoint.class.getName());
          
          Show
          yuzhihong@gmail.com Ted Yu added a comment - Took a quick look at GroupShellTest which misses master coprocessor setup similar to the following: + TEST_UTIL.getConfiguration().set(CoprocessorHost.MASTER_COPROCESSOR_CONF_KEY, + GroupAdminEndpoint.class.getName());
          Hide
          toffer Francis Liu added a comment -

          Is this really how we're supposed to run the other shell unit tests? What if there's something that needs to be changed/fixed in TestShell?

          Show
          toffer Francis Liu added a comment - Is this really how we're supposed to run the other shell unit tests? What if there's something that needs to be changed/fixed in TestShell?
          Hide
          yuzhihong@gmail.com Ted Yu added a comment -

          Francis:
          You can copy the following file from branch-1.1 to master branch so that you can run TestShell:
          hbase-shell/src/test/java/org/apache/hadoop/hbase/client/TestShell.java

          I ran TestShell locally and still got:

          test_Test_Basic_Group_Commands(Hbase::GroupShellTest):
          NativeException: org.apache.hadoop.hbase.exceptions.UnknownProtocolException: org.apache.hadoop.hbase.exceptions.UnknownProtocolException: No registered master coprocessor    service found for name hbase.pb.GroupAdminService
            at org.apache.hadoop.hbase.master.MasterRpcServices.execMasterService(MasterRpcServices.java:682)
            at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:57964)
            at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2213)
            at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
            at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
            at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
            at java.lang.Thread.run(Thread.java:745)
          
              sun/reflect/NativeConstructorAccessorImpl.java:-2:in `newInstance0'
              sun/reflect/NativeConstructorAccessorImpl.java:57:in `newInstance'
              sun/reflect/DelegatingConstructorAccessorImpl.java:45:in `newInstance'
              java/lang/reflect/Constructor.java:526:in `newInstance'
              org/apache/hadoop/ipc/RemoteException.java:106:in `instantiateException'
              org/apache/hadoop/ipc/RemoteException.java:95:in `unwrapRemoteException'
              org/apache/hadoop/hbase/ipc/AsyncCall.java:127:in `setFailed'
              org/apache/hadoop/hbase/ipc/AsyncServerResponseHandler.java:83:in `channelRead'
              io/netty/channel/AbstractChannelHandlerContext.java:308:in `invokeChannelRead'
              io/netty/channel/AbstractChannelHandlerContext.java:294:in `fireChannelRead'
              io/netty/handler/codec/ByteToMessageDecoder.java:244:in `channelRead'
              io/netty/channel/AbstractChannelHandlerContext.java:308:in `invokeChannelRead'
              io/netty/channel/AbstractChannelHandlerContext.java:294:in `fireChannelRead'
              io/netty/channel/DefaultChannelPipeline.java:846:in `fireChannelRead'
              io/netty/channel/nio/AbstractNioByteChannel.java:131:in `read'
              io/netty/channel/nio/NioEventLoop.java:511:in `processSelectedKey'
              io/netty/channel/nio/NioEventLoop.java:468:in `processSelectedKeysOptimized'
              io/netty/channel/nio/NioEventLoop.java:382:in `processSelectedKeys'
              io/netty/channel/nio/NioEventLoop.java:354:in `run'
              io/netty/util/concurrent/SingleThreadEventExecutor.java:110:in `run'
              java/lang/Thread.java:745:in `run'
              ./src/test/ruby/shell/group_shell_test.rb:43:in `test_Test_Basic_Group_Commands'
              org/jruby/RubyProc.java:270:in `call'
              org/jruby/RubyKernel.java:2105:in `send'
              org/jruby/RubyArray.java:1620:in `each'
              org/jruby/RubyArray.java:1620:in `each'
          
            2) Failure:
          test_Test_bogus_arguments(Hbase::GroupShellTest)
              [./src/test/ruby/shell/group_shell_test.rb:85:in `test_Test_bogus_arguments'
               org/jruby/RubyProc.java:270:in `call'
               org/jruby/RubyKernel.java:2105:in `send'
               org/jruby/RubyArray.java:1620:in `each'
               org/jruby/RubyArray.java:1620:in `each']:
          <ArgumentError> exception expected but was
          Class: <NativeException>
          Message: <"org.apache.hadoop.hbase.exceptions.UnknownProtocolException: org.apache.hadoop.hbase.exceptions.UnknownProtocolException: No registered master coprocessor service  found for name hbase.pb.GroupAdminService\n\tat org.apache.hadoop.hbase.master.MasterRpcServices.execMasterService(MasterRpcServices.java:682)\n\tat org.apache.hadoop.hbase.  protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:57964)\n\tat org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2213)\n\tat org.   apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)\n\tat org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)\n\tat org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)\n\tat java.lang.Thread.run(Thread.java:745)\n">
          
          Show
          yuzhihong@gmail.com Ted Yu added a comment - Francis: You can copy the following file from branch-1.1 to master branch so that you can run TestShell: hbase-shell/src/test/java/org/apache/hadoop/hbase/client/TestShell.java I ran TestShell locally and still got: test_Test_Basic_Group_Commands(Hbase::GroupShellTest): NativeException: org.apache.hadoop.hbase.exceptions.UnknownProtocolException: org.apache.hadoop.hbase.exceptions.UnknownProtocolException: No registered master coprocessor service found for name hbase.pb.GroupAdminService at org.apache.hadoop.hbase.master.MasterRpcServices.execMasterService(MasterRpcServices.java:682) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:57964) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2213) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108) at java.lang. Thread .run( Thread .java:745) sun/reflect/NativeConstructorAccessorImpl.java:-2:in `newInstance0' sun/reflect/NativeConstructorAccessorImpl.java:57:in `newInstance' sun/reflect/DelegatingConstructorAccessorImpl.java:45:in `newInstance' java/lang/reflect/Constructor.java:526:in `newInstance' org/apache/hadoop/ipc/RemoteException.java:106:in `instantiateException' org/apache/hadoop/ipc/RemoteException.java:95:in `unwrapRemoteException' org/apache/hadoop/hbase/ipc/AsyncCall.java:127:in `setFailed' org/apache/hadoop/hbase/ipc/AsyncServerResponseHandler.java:83:in `channelRead' io/netty/channel/AbstractChannelHandlerContext.java:308:in `invokeChannelRead' io/netty/channel/AbstractChannelHandlerContext.java:294:in `fireChannelRead' io/netty/handler/codec/ByteToMessageDecoder.java:244:in `channelRead' io/netty/channel/AbstractChannelHandlerContext.java:308:in `invokeChannelRead' io/netty/channel/AbstractChannelHandlerContext.java:294:in `fireChannelRead' io/netty/channel/DefaultChannelPipeline.java:846:in `fireChannelRead' io/netty/channel/nio/AbstractNioByteChannel.java:131:in `read' io/netty/channel/nio/NioEventLoop.java:511:in `processSelectedKey' io/netty/channel/nio/NioEventLoop.java:468:in `processSelectedKeysOptimized' io/netty/channel/nio/NioEventLoop.java:382:in `processSelectedKeys' io/netty/channel/nio/NioEventLoop.java:354:in `run' io/netty/util/concurrent/SingleThreadEventExecutor.java:110:in `run' java/lang/ Thread .java:745:in `run' ./src/test/ruby/shell/group_shell_test.rb:43:in `test_Test_Basic_Group_Commands' org/jruby/RubyProc.java:270:in `call' org/jruby/RubyKernel.java:2105:in `send' org/jruby/RubyArray.java:1620:in `each' org/jruby/RubyArray.java:1620:in `each' 2) Failure: test_Test_bogus_arguments(Hbase::GroupShellTest) [./src/test/ruby/shell/group_shell_test.rb:85:in `test_Test_bogus_arguments' org/jruby/RubyProc.java:270:in `call' org/jruby/RubyKernel.java:2105:in `send' org/jruby/RubyArray.java:1620:in `each' org/jruby/RubyArray.java:1620:in `each']: <ArgumentError> exception expected but was Class : <NativeException> Message: < "org.apache.hadoop.hbase.exceptions.UnknownProtocolException: org.apache.hadoop.hbase.exceptions.UnknownProtocolException: No registered master coprocessor service found for name hbase.pb.GroupAdminService\n\tat org.apache.hadoop.hbase.master.MasterRpcServices.execMasterService(MasterRpcServices.java:682)\n\tat org.apache.hadoop.hbase. protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:57964)\n\tat org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2213)\n\tat org. apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)\n\tat org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)\n\tat org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)\n\tat java.lang. Thread .run( Thread .java:745)\n" >
          Hide
          yuzhihong@gmail.com Ted Yu added a comment -

          From https://builds.apache.org/job/PreCommit-HBASE-Build/16827/consoleFull :

          testKillRS(org.apache.hadoop.hbase.group.TestGroups)  Time elapsed: 10.725 sec  <<< FAILURE!
          java.lang.AssertionError: expected:<1> but was:<0>
          	at org.junit.Assert.fail(Assert.java:88)
          	at org.junit.Assert.failNotEquals(Assert.java:834)
          	at org.junit.Assert.assertEquals(Assert.java:645)
          	at org.junit.Assert.assertEquals(Assert.java:631)
          	at org.apache.hadoop.hbase.group.TestGroupsBase.testKillRS(TestGroupsBase.java:612)
          

          I am running TestShell locally to see if it passes.

          Show
          yuzhihong@gmail.com Ted Yu added a comment - From https://builds.apache.org/job/PreCommit-HBASE-Build/16827/consoleFull : testKillRS(org.apache.hadoop.hbase.group.TestGroups) Time elapsed: 10.725 sec <<< FAILURE! java.lang.AssertionError: expected:<1> but was:<0> at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:834) at org.junit.Assert.assertEquals(Assert.java:645) at org.junit.Assert.assertEquals(Assert.java:631) at org.apache.hadoop.hbase.group.TestGroupsBase.testKillRS(TestGroupsBase.java:612) I am running TestShell locally to see if it passes.
          Hide
          toffer Francis Liu added a comment -

          Looks like Ted Yu is OOO.

          Enis Soztutar Elliott Clark Andrew Purtell Any other feedback/concerns? Anything else I should address?

          Show
          toffer Francis Liu added a comment - Looks like Ted Yu is OOO. Enis Soztutar Elliott Clark Andrew Purtell Any other feedback/concerns? Anything else I should address?
          Hide
          toffer Francis Liu added a comment -

          Ted Yu Looks like fixing the normalizer made the core tests pass, I'm not able to reproduce the shell failure you presented?

          Show
          toffer Francis Liu added a comment - Ted Yu Looks like fixing the normalizer made the core tests pass, I'm not able to reproduce the shell failure you presented?
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12776745/hbase-6721-v19.patch
          against master branch at commit 9647fee3f0f196d064879afd41b9eff51d5aa036.
          ATTACHMENT ID: 12776745

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 45 new or modified tests.

          +1 hadoop versions. The patch compiles with all supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0 2.7.1)

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 protoc. The applied patch does not increase the total number of protoc compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 checkstyle. The applied patch generated new checkstyle errors. Check build console for list of new errors.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 lineLengths. The patch introduces the following lines longer than 100:
          + * <code>rpc GetGroupInfo(.hbase.pb.GetGroupInfoRequest) returns (.hbase.pb.GetGroupInfoResponse);</code>
          + * <code>rpc GetGroupInfoOfTable(.hbase.pb.GetGroupInfoOfTableRequest) returns (.hbase.pb.GetGroupInfoOfTableResponse);</code>
          + * <code>rpc GetGroupInfoOfServer(.hbase.pb.GetGroupInfoOfServerRequest) returns (.hbase.pb.GetGroupInfoOfServerResponse);</code>
          + * <code>rpc MoveServers(.hbase.pb.MoveServersRequest) returns (.hbase.pb.MoveServersResponse);</code>
          + * <code>rpc MoveTables(.hbase.pb.MoveTablesRequest) returns (.hbase.pb.MoveTablesResponse);</code>
          + * <code>rpc RemoveGroup(.hbase.pb.RemoveGroupRequest) returns (.hbase.pb.RemoveGroupResponse);</code>
          + * <code>rpc BalanceGroup(.hbase.pb.BalanceGroupRequest) returns (.hbase.pb.BalanceGroupResponse);</code>
          + * <code>rpc ListGroupInfos(.hbase.pb.ListGroupInfosRequest) returns (.hbase.pb.ListGroupInfosResponse);</code>
          + * <code>rpc GetGroupInfo(.hbase.pb.GetGroupInfoRequest) returns (.hbase.pb.GetGroupInfoResponse);</code>
          + * <code>rpc GetGroupInfoOfTable(.hbase.pb.GetGroupInfoOfTableRequest) returns (.hbase.pb.GetGroupInfoOfTableResponse);</code>

          +1 site. The mvn post-site goal succeeds with this patch.

          +1 core tests. The patch passed unit tests in .

          +1 zombies. No zombie tests found running at the end of the build.

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/16827//testReport/
          Release Findbugs (version 2.0.3) warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/16827//artifact/patchprocess/newFindbugsWarnings.html
          Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/16827//artifact/patchprocess/checkstyle-aggregate.html

          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/16827//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12776745/hbase-6721-v19.patch against master branch at commit 9647fee3f0f196d064879afd41b9eff51d5aa036. ATTACHMENT ID: 12776745 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 45 new or modified tests. +1 hadoop versions . The patch compiles with all supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0 2.7.1) +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 protoc . The applied patch does not increase the total number of protoc compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. -1 checkstyle . The applied patch generated new checkstyle errors. Check build console for list of new errors. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 lineLengths . The patch introduces the following lines longer than 100: + * <code>rpc GetGroupInfo(.hbase.pb.GetGroupInfoRequest) returns (.hbase.pb.GetGroupInfoResponse);</code> + * <code>rpc GetGroupInfoOfTable(.hbase.pb.GetGroupInfoOfTableRequest) returns (.hbase.pb.GetGroupInfoOfTableResponse);</code> + * <code>rpc GetGroupInfoOfServer(.hbase.pb.GetGroupInfoOfServerRequest) returns (.hbase.pb.GetGroupInfoOfServerResponse);</code> + * <code>rpc MoveServers(.hbase.pb.MoveServersRequest) returns (.hbase.pb.MoveServersResponse);</code> + * <code>rpc MoveTables(.hbase.pb.MoveTablesRequest) returns (.hbase.pb.MoveTablesResponse);</code> + * <code>rpc RemoveGroup(.hbase.pb.RemoveGroupRequest) returns (.hbase.pb.RemoveGroupResponse);</code> + * <code>rpc BalanceGroup(.hbase.pb.BalanceGroupRequest) returns (.hbase.pb.BalanceGroupResponse);</code> + * <code>rpc ListGroupInfos(.hbase.pb.ListGroupInfosRequest) returns (.hbase.pb.ListGroupInfosResponse);</code> + * <code>rpc GetGroupInfo(.hbase.pb.GetGroupInfoRequest) returns (.hbase.pb.GetGroupInfoResponse);</code> + * <code>rpc GetGroupInfoOfTable(.hbase.pb.GetGroupInfoOfTableRequest) returns (.hbase.pb.GetGroupInfoOfTableResponse);</code> +1 site . The mvn post-site goal succeeds with this patch. +1 core tests . The patch passed unit tests in . +1 zombies . No zombie tests found running at the end of the build. Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/16827//testReport/ Release Findbugs (version 2.0.3) warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/16827//artifact/patchprocess/newFindbugsWarnings.html Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/16827//artifact/patchprocess/checkstyle-aggregate.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/16827//console This message is automatically generated.
          Hide
          toffer Francis Liu added a comment -

          Patch fixes normalizer unit test failure.

          Show
          toffer Francis Liu added a comment - Patch fixes normalizer unit test failure.
          Hide
          toffer Francis Liu added a comment -

          Ted Yu Looks like TestSimpleRegionNormalizer is using negative port numbers for ServerName. Should be a simple fix to update the test. How are you able to run TestShell? I only see an abstract class.

          Show
          toffer Francis Liu added a comment - Ted Yu Looks like TestSimpleRegionNormalizer is using negative port numbers for ServerName. Should be a simple fix to update the test. How are you able to run TestShell? I only see an abstract class.
          Hide
          yuzhihong@gmail.com Ted Yu added a comment -

          Test failure in TestSimpleRegionNormalizer is reproducible:

          testNoNormalizationForMetaTable(org.apache.hadoop.hbase.master.normalizer.TestSimpleRegionNormalizer)  Time elapsed: 0.337 sec  <<< ERROR!
          java.lang.IllegalArgumentException: null
            at com.google.common.base.Preconditions.checkArgument(Preconditions.java:76)
            at com.google.common.net.HostAndPort.fromParts(HostAndPort.java:130)
            at org.apache.hadoop.hbase.ServerName.<init>(ServerName.java:105)
            at org.apache.hadoop.hbase.ServerName.valueOf(ServerName.java:158)
            at org.apache.hadoop.hbase.master.normalizer.TestSimpleRegionNormalizer.setupMocksForNormalizer(TestSimpleRegionNormalizer.java:223)
            at org.apache.hadoop.hbase.master.normalizer.TestSimpleRegionNormalizer.testNoNormalizationForMetaTable(TestSimpleRegionNormalizer.java:71)
          
          testNoNormalizationIfTooFewRegions(org.apache.hadoop.hbase.master.normalizer.TestSimpleRegionNormalizer)  Time elapsed: 0.045 sec  <<< ERROR!
          java.lang.IllegalArgumentException: null
            at com.google.common.base.Preconditions.checkArgument(Preconditions.java:76)
            at com.google.common.net.HostAndPort.fromParts(HostAndPort.java:130)
            at org.apache.hadoop.hbase.ServerName.<init>(ServerName.java:105)
            at org.apache.hadoop.hbase.ServerName.valueOf(ServerName.java:158)
            at org.apache.hadoop.hbase.master.normalizer.TestSimpleRegionNormalizer.setupMocksForNormalizer(TestSimpleRegionNormalizer.java:223)
            at org.apache.hadoop.hbase.master.normalizer.TestSimpleRegionNormalizer.testNoNormalizationIfTooFewRegions(TestSimpleRegionNormalizer.java:90)
          
          Show
          yuzhihong@gmail.com Ted Yu added a comment - Test failure in TestSimpleRegionNormalizer is reproducible: testNoNormalizationForMetaTable(org.apache.hadoop.hbase.master.normalizer.TestSimpleRegionNormalizer) Time elapsed: 0.337 sec <<< ERROR! java.lang.IllegalArgumentException: null at com.google.common.base.Preconditions.checkArgument(Preconditions.java:76) at com.google.common.net.HostAndPort.fromParts(HostAndPort.java:130) at org.apache.hadoop.hbase.ServerName.<init>(ServerName.java:105) at org.apache.hadoop.hbase.ServerName.valueOf(ServerName.java:158) at org.apache.hadoop.hbase.master.normalizer.TestSimpleRegionNormalizer.setupMocksForNormalizer(TestSimpleRegionNormalizer.java:223) at org.apache.hadoop.hbase.master.normalizer.TestSimpleRegionNormalizer.testNoNormalizationForMetaTable(TestSimpleRegionNormalizer.java:71) testNoNormalizationIfTooFewRegions(org.apache.hadoop.hbase.master.normalizer.TestSimpleRegionNormalizer) Time elapsed: 0.045 sec <<< ERROR! java.lang.IllegalArgumentException: null at com.google.common.base.Preconditions.checkArgument(Preconditions.java:76) at com.google.common.net.HostAndPort.fromParts(HostAndPort.java:130) at org.apache.hadoop.hbase.ServerName.<init>(ServerName.java:105) at org.apache.hadoop.hbase.ServerName.valueOf(ServerName.java:158) at org.apache.hadoop.hbase.master.normalizer.TestSimpleRegionNormalizer.setupMocksForNormalizer(TestSimpleRegionNormalizer.java:223) at org.apache.hadoop.hbase.master.normalizer.TestSimpleRegionNormalizer.testNoNormalizationIfTooFewRegions(TestSimpleRegionNormalizer.java:90)
          Hide
          yuzhihong@gmail.com Ted Yu added a comment -

          Running TestShell with latest patch, I got:

            1) Error:
          test_Test_Basic_Group_Commands(Hbase::GroupShellTest):
          NativeException: org.apache.hadoop.hbase.exceptions.UnknownProtocolException: org.apache.hadoop.hbase.exceptions.UnknownProtocolException: No registered master coprocessor     service found for name hbase.pb.GroupAdminService
            at org.apache.hadoop.hbase.master.MasterRpcServices.execMasterService(MasterRpcServices.java:682)
            at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:57964)
            at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2184)
            at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
            at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
            at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
          ...
            2) Failure:
          test_Test_bogus_arguments(Hbase::GroupShellTest)
              [./src/test/ruby/shell/group_shell_test.rb:85:in `test_Test_bogus_arguments'
               org/jruby/RubyProc.java:270:in `call'
               org/jruby/RubyKernel.java:2105:in `send'
               org/jruby/RubyArray.java:1620:in `each'
               org/jruby/RubyArray.java:1620:in `each']:
          <ArgumentError> exception expected but was
          Class: <NativeException>
          Message: <"org.apache.hadoop.hbase.exceptions.UnknownProtocolException: org.apache.hadoop.hbase.exceptions.UnknownProtocolException: No registered master coprocessor service   found for name hbase.pb.GroupAdminService\n\tat org.apache.hadoop.hbase.master.MasterRpcServices.execMasterService(MasterRpcServices.java:682)\n\tat org.apache.hadoop.hbase.   protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:57964)\n\tat org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2184)\n\tat org.    apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)\n\tat org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)\n\tat org.apache.hadoop.hbase.ipc. RpcExecutor$1.run(RpcExecutor.java:108)\n\tat java.lang.Thread.run(Thread.java:745)\n">
          

          Francis:
          Do the tests pass for you ?

          Show
          yuzhihong@gmail.com Ted Yu added a comment - Running TestShell with latest patch, I got: 1) Error: test_Test_Basic_Group_Commands(Hbase::GroupShellTest): NativeException: org.apache.hadoop.hbase.exceptions.UnknownProtocolException: org.apache.hadoop.hbase.exceptions.UnknownProtocolException: No registered master coprocessor service found for name hbase.pb.GroupAdminService at org.apache.hadoop.hbase.master.MasterRpcServices.execMasterService(MasterRpcServices.java:682) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:57964) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2184) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108) ... 2) Failure: test_Test_bogus_arguments(Hbase::GroupShellTest) [./src/test/ruby/shell/group_shell_test.rb:85:in `test_Test_bogus_arguments' org/jruby/RubyProc.java:270:in `call' org/jruby/RubyKernel.java:2105:in `send' org/jruby/RubyArray.java:1620:in `each' org/jruby/RubyArray.java:1620:in `each']: <ArgumentError> exception expected but was Class : <NativeException> Message: < "org.apache.hadoop.hbase.exceptions.UnknownProtocolException: org.apache.hadoop.hbase.exceptions.UnknownProtocolException: No registered master coprocessor service found for name hbase.pb.GroupAdminService\n\tat org.apache.hadoop.hbase.master.MasterRpcServices.execMasterService(MasterRpcServices.java:682)\n\tat org.apache.hadoop.hbase. protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:57964)\n\tat org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2184)\n\tat org. apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)\n\tat org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)\n\tat org.apache.hadoop.hbase.ipc. RpcExecutor$1.run(RpcExecutor.java:108)\n\tat java.lang. Thread .run( Thread .java:745)\n" > Francis: Do the tests pass for you ?
          Hide
          enis Enis Soztutar added a comment -

          Thanks Francis Liu for the updated patch. All of the review items have been addressed, and hence I'm +1 for committing this.

          Andrew Purtell, Elliott Clark or others have any more concerns, feedback? Given that this is an optional coprocessor based thing, I think this can directly go to master.

          Show
          enis Enis Soztutar added a comment - Thanks Francis Liu for the updated patch. All of the review items have been addressed, and hence I'm +1 for committing this. Andrew Purtell , Elliott Clark or others have any more concerns, feedback? Given that this is an optional coprocessor based thing, I think this can directly go to master.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12773431/hbase-6721-v18.patch
          against master branch at commit ea48ef86512addc3dc9bcde4b7433a3ac5881424.
          ATTACHMENT ID: 12773431

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 42 new or modified tests.

          +1 hadoop versions. The patch compiles with all supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0 2.7.1)

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 protoc. The applied patch does not increase the total number of protoc compiler warnings.

          -1 javadoc. The javadoc tool appears to have generated 5 warning messages.

          -1 checkstyle. The applied patch generated 1752 checkstyle errors (more than the master's current 1727 errors).

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 lineLengths. The patch introduces the following lines longer than 100:
          + * <code>rpc GetGroupInfo(.hbase.pb.GetGroupInfoRequest) returns (.hbase.pb.GetGroupInfoResponse);</code>
          + * <code>rpc GetGroupInfoOfTable(.hbase.pb.GetGroupInfoOfTableRequest) returns (.hbase.pb.GetGroupInfoOfTableResponse);</code>
          + * <code>rpc GetGroupInfoOfServer(.hbase.pb.GetGroupInfoOfServerRequest) returns (.hbase.pb.GetGroupInfoOfServerResponse);</code>
          + * <code>rpc MoveServers(.hbase.pb.MoveServersRequest) returns (.hbase.pb.MoveServersResponse);</code>
          + * <code>rpc MoveTables(.hbase.pb.MoveTablesRequest) returns (.hbase.pb.MoveTablesResponse);</code>
          + * <code>rpc RemoveGroup(.hbase.pb.RemoveGroupRequest) returns (.hbase.pb.RemoveGroupResponse);</code>
          + * <code>rpc BalanceGroup(.hbase.pb.BalanceGroupRequest) returns (.hbase.pb.BalanceGroupResponse);</code>
          + * <code>rpc ListGroupInfos(.hbase.pb.ListGroupInfosRequest) returns (.hbase.pb.ListGroupInfosResponse);</code>
          + * <code>rpc GetGroupInfo(.hbase.pb.GetGroupInfoRequest) returns (.hbase.pb.GetGroupInfoResponse);</code>
          + * <code>rpc GetGroupInfoOfTable(.hbase.pb.GetGroupInfoOfTableRequest) returns (.hbase.pb.GetGroupInfoOfTableResponse);</code>

          +1 site. The mvn post-site goal succeeds with this patch.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.master.normalizer.TestSimpleRegionNormalizer

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/16603//testReport/
          Release Findbugs (version 2.0.3) warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/16603//artifact/patchprocess/newFindbugsWarnings.html
          Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/16603//artifact/patchprocess/checkstyle-aggregate.html

          Javadoc warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/16603//artifact/patchprocess/patchJavadocWarnings.txt
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/16603//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12773431/hbase-6721-v18.patch against master branch at commit ea48ef86512addc3dc9bcde4b7433a3ac5881424. ATTACHMENT ID: 12773431 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 42 new or modified tests. +1 hadoop versions . The patch compiles with all supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0 2.7.1) +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 protoc . The applied patch does not increase the total number of protoc compiler warnings. -1 javadoc . The javadoc tool appears to have generated 5 warning messages. -1 checkstyle . The applied patch generated 1752 checkstyle errors (more than the master's current 1727 errors). +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 lineLengths . The patch introduces the following lines longer than 100: + * <code>rpc GetGroupInfo(.hbase.pb.GetGroupInfoRequest) returns (.hbase.pb.GetGroupInfoResponse);</code> + * <code>rpc GetGroupInfoOfTable(.hbase.pb.GetGroupInfoOfTableRequest) returns (.hbase.pb.GetGroupInfoOfTableResponse);</code> + * <code>rpc GetGroupInfoOfServer(.hbase.pb.GetGroupInfoOfServerRequest) returns (.hbase.pb.GetGroupInfoOfServerResponse);</code> + * <code>rpc MoveServers(.hbase.pb.MoveServersRequest) returns (.hbase.pb.MoveServersResponse);</code> + * <code>rpc MoveTables(.hbase.pb.MoveTablesRequest) returns (.hbase.pb.MoveTablesResponse);</code> + * <code>rpc RemoveGroup(.hbase.pb.RemoveGroupRequest) returns (.hbase.pb.RemoveGroupResponse);</code> + * <code>rpc BalanceGroup(.hbase.pb.BalanceGroupRequest) returns (.hbase.pb.BalanceGroupResponse);</code> + * <code>rpc ListGroupInfos(.hbase.pb.ListGroupInfosRequest) returns (.hbase.pb.ListGroupInfosResponse);</code> + * <code>rpc GetGroupInfo(.hbase.pb.GetGroupInfoRequest) returns (.hbase.pb.GetGroupInfoResponse);</code> + * <code>rpc GetGroupInfoOfTable(.hbase.pb.GetGroupInfoOfTableRequest) returns (.hbase.pb.GetGroupInfoOfTableResponse);</code> +1 site . The mvn post-site goal succeeds with this patch. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.master.normalizer.TestSimpleRegionNormalizer Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/16603//testReport/ Release Findbugs (version 2.0.3) warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/16603//artifact/patchprocess/newFindbugsWarnings.html Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/16603//artifact/patchprocess/checkstyle-aggregate.html Javadoc warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/16603//artifact/patchprocess/patchJavadocWarnings.txt Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/16603//console This message is automatically generated.
          Hide
          toffer Francis Liu added a comment -

          Also moved unassign of misplaced regions out of the balancer and added unit test.

          Updated RB with both patches:

          https://reviews.apache.org/r/38224/

          Show
          toffer Francis Liu added a comment - Also moved unassign of misplaced regions out of the balancer and added unit test. Updated RB with both patches: https://reviews.apache.org/r/38224/
          Hide
          toffer Francis Liu added a comment -

          Added v18 patch which addresses metnioned comments. Also rebased to HEAD and fixed issues.

          Show
          toffer Francis Liu added a comment - Added v18 patch which addresses metnioned comments. Also rebased to HEAD and fixed issues.
          Hide
          enis Enis Soztutar added a comment -

          Thanks Francis Liu. The list sounds about right.

          With the next update, I can now buy the patch a beer (in some states).

          We can all grab a (virtual) beer once this is committed.

          Show
          enis Enis Soztutar added a comment - Thanks Francis Liu . The list sounds about right. With the next update, I can now buy the patch a beer (in some states). We can all grab a (virtual) beer once this is committed.
          Hide
          toffer Francis Liu added a comment -

          Thanks for the comprehensive testing and fixes Enis Soztutar. Let me try and sum up the outstanding things that need to be done. Let me know if I missed anything:

          1. Unit test for shell commands
          2. Fix rsgroup table name description
          3. "NPE in group shell commands with nonexisting groups"
          4. "Group names should be restricted to alphanumeric only. "
          5. "move_group_servers and move_group_tables arguments are listed as 1, although should be 2:"
          6. File Jira for UI, do it as follow-up
          7. File Jira for documentation, do it as follow-up.

          8. I was thinking of testing to remove the default group, but did not do it yet. Reading of the code seems to allow it. We should fix that.

          This is being checked in GroupInfomManagerImpl.removeGroup so this is covered.

          With the next update, I can now buy the patch a beer (in some states).

          Show
          toffer Francis Liu added a comment - Thanks for the comprehensive testing and fixes Enis Soztutar . Let me try and sum up the outstanding things that need to be done. Let me know if I missed anything: 1. Unit test for shell commands 2. Fix rsgroup table name description 3. "NPE in group shell commands with nonexisting groups" 4. "Group names should be restricted to alphanumeric only. " 5. "move_group_servers and move_group_tables arguments are listed as 1, although should be 2:" 6. File Jira for UI, do it as follow-up 7. File Jira for documentation, do it as follow-up. 8. I was thinking of testing to remove the default group, but did not do it yet. Reading of the code seems to allow it. We should fix that. This is being checked in GroupInfomManagerImpl.removeGroup so this is covered. With the next update, I can now buy the patch a beer (in some states).
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12770514/hbase-6721-v17.patch
          against master branch at commit ceddec9141b658c790d2fb995c970982ab082028.
          ATTACHMENT ID: 12770514

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 36 new or modified tests.

          -1 javac. The patch appears to cause mvn compile goal to fail with Hadoop version 2.4.0.

          Compilation errors resume:
          [ERROR] COMPILATION ERROR :
          [ERROR] /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/group/GroupInfoManagerImpl.java:[588,47] cannot find symbol
          [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.2:compile (default-compile) on project hbase-server: Compilation failure
          [ERROR] /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/group/GroupInfoManagerImpl.java:[588,47] cannot find symbol
          [ERROR] symbol: method getHRegionInfo(org.apache.hadoop.hbase.client.Result)
          [ERROR] location: class org.apache.hadoop.hbase.HRegionInfo
          [ERROR] -> [Help 1]
          [ERROR]
          [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
          [ERROR] Re-run Maven using the -X switch to enable full debug logging.
          [ERROR]
          [ERROR] For more information about the errors and possible solutions, please read the following articles:
          [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
          [ERROR]
          [ERROR] After correcting the problems, you can resume the build with the command
          [ERROR] mvn <goals> -rf :hbase-server

          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/16380//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12770514/hbase-6721-v17.patch against master branch at commit ceddec9141b658c790d2fb995c970982ab082028. ATTACHMENT ID: 12770514 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 36 new or modified tests. -1 javac . The patch appears to cause mvn compile goal to fail with Hadoop version 2.4.0. Compilation errors resume: [ERROR] COMPILATION ERROR : [ERROR] /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/group/GroupInfoManagerImpl.java: [588,47] cannot find symbol [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.2:compile (default-compile) on project hbase-server: Compilation failure [ERROR] /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/group/GroupInfoManagerImpl.java: [588,47] cannot find symbol [ERROR] symbol: method getHRegionInfo(org.apache.hadoop.hbase.client.Result) [ERROR] location: class org.apache.hadoop.hbase.HRegionInfo [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn <goals> -rf :hbase-server Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/16380//console This message is automatically generated.
          Hide
          enis Enis Soztutar added a comment -

          Here is a v17 patch which has these changes on top of v16:

          • Addresses item (4) above. Fixed get_server_group and get_table_group commands.
          • Some ruby changes
          • Changed the unit tests so that they will work the same on master code or branch-1 code (due to colocated master changes).
          • Fixed a timing issue with the test causing the IT to fail.

          Now, I am able to run the IntegrationTestGroup in a 7 node cluster, and the unit test works.

          Francis Liu I think there are only a couple of remaining items left. Let me know, if you need help in addressing some of them.

          Show
          enis Enis Soztutar added a comment - Here is a v17 patch which has these changes on top of v16: Addresses item (4) above. Fixed get_server_group and get_table_group commands. Some ruby changes Changed the unit tests so that they will work the same on master code or branch-1 code (due to colocated master changes). Fixed a timing issue with the test causing the IT to fail. Now, I am able to run the IntegrationTestGroup in a 7 node cluster, and the unit test works. Francis Liu I think there are only a couple of remaining items left. Let me know, if you need help in addressing some of them.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12769912/hbase-6721-v16.patch
          against master branch at commit 683f84e6a217dfd872e5f1be82c7aa4cdf232ec1.
          ATTACHMENT ID: 12769912

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 33 new or modified tests.

          +1 hadoop versions. The patch compiles with all supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0 2.7.1)

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 protoc. The applied patch does not increase the total number of protoc compiler warnings.

          -1 javadoc. The javadoc tool appears to have generated 4 warning messages.

          -1 checkstyle. The applied patch generated 1755 checkstyle errors (more than the master's current 1730 errors).

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 lineLengths. The patch introduces the following lines longer than 100:
          + * <code>rpc GetGroupInfo(.hbase.pb.GetGroupInfoRequest) returns (.hbase.pb.GetGroupInfoResponse);</code>
          + * <code>rpc GetGroupInfoOfTable(.hbase.pb.GetGroupInfoOfTableRequest) returns (.hbase.pb.GetGroupInfoOfTableResponse);</code>
          + * <code>rpc GetGroupInfoOfServer(.hbase.pb.GetGroupInfoOfServerRequest) returns (.hbase.pb.GetGroupInfoOfServerResponse);</code>
          + * <code>rpc MoveServers(.hbase.pb.MoveServersRequest) returns (.hbase.pb.MoveServersResponse);</code>
          + * <code>rpc MoveTables(.hbase.pb.MoveTablesRequest) returns (.hbase.pb.MoveTablesResponse);</code>
          + * <code>rpc RemoveGroup(.hbase.pb.RemoveGroupRequest) returns (.hbase.pb.RemoveGroupResponse);</code>
          + * <code>rpc BalanceGroup(.hbase.pb.BalanceGroupRequest) returns (.hbase.pb.BalanceGroupResponse);</code>
          + * <code>rpc ListGroupInfos(.hbase.pb.ListGroupInfosRequest) returns (.hbase.pb.ListGroupInfosResponse);</code>
          + * <code>rpc GetGroupInfo(.hbase.pb.GetGroupInfoRequest) returns (.hbase.pb.GetGroupInfoResponse);</code>
          + * <code>rpc GetGroupInfoOfTable(.hbase.pb.GetGroupInfoOfTableRequest) returns (.hbase.pb.GetGroupInfoOfTableResponse);</code>

          +1 site. The mvn post-site goal succeeds with this patch.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.client.TestAsyncProcess

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/16327//testReport/
          Release Findbugs (version 2.0.3) warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/16327//artifact/patchprocess/newFindbugsWarnings.html
          Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/16327//artifact/patchprocess/checkstyle-aggregate.html

          Javadoc warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/16327//artifact/patchprocess/patchJavadocWarnings.txt
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/16327//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12769912/hbase-6721-v16.patch against master branch at commit 683f84e6a217dfd872e5f1be82c7aa4cdf232ec1. ATTACHMENT ID: 12769912 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 33 new or modified tests. +1 hadoop versions . The patch compiles with all supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0 2.7.1) +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 protoc . The applied patch does not increase the total number of protoc compiler warnings. -1 javadoc . The javadoc tool appears to have generated 4 warning messages. -1 checkstyle . The applied patch generated 1755 checkstyle errors (more than the master's current 1730 errors). +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 lineLengths . The patch introduces the following lines longer than 100: + * <code>rpc GetGroupInfo(.hbase.pb.GetGroupInfoRequest) returns (.hbase.pb.GetGroupInfoResponse);</code> + * <code>rpc GetGroupInfoOfTable(.hbase.pb.GetGroupInfoOfTableRequest) returns (.hbase.pb.GetGroupInfoOfTableResponse);</code> + * <code>rpc GetGroupInfoOfServer(.hbase.pb.GetGroupInfoOfServerRequest) returns (.hbase.pb.GetGroupInfoOfServerResponse);</code> + * <code>rpc MoveServers(.hbase.pb.MoveServersRequest) returns (.hbase.pb.MoveServersResponse);</code> + * <code>rpc MoveTables(.hbase.pb.MoveTablesRequest) returns (.hbase.pb.MoveTablesResponse);</code> + * <code>rpc RemoveGroup(.hbase.pb.RemoveGroupRequest) returns (.hbase.pb.RemoveGroupResponse);</code> + * <code>rpc BalanceGroup(.hbase.pb.BalanceGroupRequest) returns (.hbase.pb.BalanceGroupResponse);</code> + * <code>rpc ListGroupInfos(.hbase.pb.ListGroupInfosRequest) returns (.hbase.pb.ListGroupInfosResponse);</code> + * <code>rpc GetGroupInfo(.hbase.pb.GetGroupInfoRequest) returns (.hbase.pb.GetGroupInfoResponse);</code> + * <code>rpc GetGroupInfoOfTable(.hbase.pb.GetGroupInfoOfTableRequest) returns (.hbase.pb.GetGroupInfoOfTableResponse);</code> +1 site . The mvn post-site goal succeeds with this patch. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.client.TestAsyncProcess Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/16327//testReport/ Release Findbugs (version 2.0.3) warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/16327//artifact/patchprocess/newFindbugsWarnings.html Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/16327//artifact/patchprocess/checkstyle-aggregate.html Javadoc warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/16327//artifact/patchprocess/patchJavadocWarnings.txt Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/16327//console This message is automatically generated.
          Hide
          enis Enis Soztutar added a comment -

          Here is a master patch that is:

          • rebased to latest master.
          • Addresses (1), (10), (11), and (12) as above.
          Show
          enis Enis Soztutar added a comment - Here is a master patch that is: rebased to latest master. Addresses (1), (10), (11), and (12) as above.
          Hide
          enis Enis Soztutar added a comment -

          We may have a problem with IntegrationTestGroup.testMoveServers.
          12. Also the test leaves the balancer turned off, which should be be the case.

          Show
          enis Enis Soztutar added a comment - We may have a problem with IntegrationTestGroup.testMoveServers. 12. Also the test leaves the balancer turned off, which should be be the case.
          Hide
          enis Enis Soztutar added a comment -

          Even more testing:
          Test 4: Create namespace with hbase.rsgroup.name pointing to a group, then create a table in that namespace. Works as expected, the new table created in correct group

          Run the integration test on the 7 node cluster:
          hbase org.apache.hadoop.hbase.IntegrationTestsDriver -regex IntegrationTestGroup
          Failed. Inspecting.

          8. I was thinking of testing to remove the default group, but did not do it yet. Reading of the code seems to allow it. We should fix that.
          9. We can add a description to the rsgroup table (minor). The master UI displays the description.
          10. In IntegrationTestGroup.java, the wait condition should not wait for exactly NUM_SLAVES_BASE, there can be more servers that NUM_SLAVES_BASE in an actual cluster.
          11. Once balancer is turned off, there is no way to turn it on since GroupAdminEndpoint.preBalanceSwitch() always returns false.

          Show
          enis Enis Soztutar added a comment - Even more testing: Test 4: Create namespace with hbase.rsgroup.name pointing to a group, then create a table in that namespace. Works as expected, the new table created in correct group Run the integration test on the 7 node cluster: hbase org.apache.hadoop.hbase.IntegrationTestsDriver -regex IntegrationTestGroup Failed. Inspecting. 8. I was thinking of testing to remove the default group, but did not do it yet. Reading of the code seems to allow it. We should fix that. 9. We can add a description to the rsgroup table (minor). The master UI displays the description. 10. In IntegrationTestGroup.java, the wait condition should not wait for exactly NUM_SLAVES_BASE, there can be more servers that NUM_SLAVES_BASE in an actual cluster. 11. Once balancer is turned off, there is no way to turn it on since GroupAdminEndpoint.preBalanceSwitch() always returns false.
          Hide
          enis Enis Soztutar added a comment -

          Finally got around testing the v15 patch on 1.1 code base with a 7 node cluster. Here are my test notes. Nothing too concerning, but we have to address some of these in the patch. This is the configuration to add to enable groups:

              <property>
                <name>hbase.coprocessor.master.classes</name>
                <value>org.apache.hadoop.hbase.group.GroupAdminEndpoint</value>
              </property>
              <property>
                <name>hbase.master.loadbalancer.class</name>
                <value>org.apache.hadoop.hbase.group.GroupBasedLoadBalancer</value>
              </property>
          

          1. Need to add this diff, so that new PB files get compiled with -Pcompile-protobuf command:

          diff --git hbase-protocol/pom.xml hbase-protocol/pom.xml
          index 8034576..d352373 100644
          --- hbase-protocol/pom.xml
          +++ hbase-protocol/pom.xml
          @@ -180,6 +180,8 @@
                                     <include>ErrorHandling.proto</include>
                                     <include>Filter.proto</include>
                                     <include>FS.proto</include>
          +                          <include>Group.proto</include>
          +                          <include>GroupAdmin.proto</include>
                                     <include>HBase.proto</include>
                                     <include>HFile.proto</include>
                                     <include>LoadBalancer.proto</include>
          

          2. NPE in group shell commands with nonexisting groups:

          hbase(main):015:0* balance_group 'nonexisting' 
          
          ERROR: java.io.IOException
          	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2156)
          	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
          	at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
          	at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
          	at java.lang.Thread.run(Thread.java:745)
          Caused by: java.lang.NullPointerException
          	at org.apache.hadoop.hbase.group.GroupAdminServer.groupGetRegionsInTransition(GroupAdminServer.java:412)
          	at org.apache.hadoop.hbase.group.GroupAdminServer.balanceGroup(GroupAdminServer.java:348)
          	at org.apache.hadoop.hbase.group.GroupAdminEndpoint.balanceGroup(GroupAdminEndpoint.java:229)
          	at org.apache.hadoop.hbase.protobuf.generated.GroupAdminProtos$GroupAdminService.callMethod(GroupAdminProtos.java:11156)
          	at org.apache.hadoop.hbase.master.MasterRpcServices.execMasterService(MasterRpcServices.java:666)
          	at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:51121)
          	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
          

          and

          hbase(main):030:0> get_group 'nonexisting'
          GROUP INFORMATION                                                                                                                                                                                                                                                  
          Servers:                                                                                                                                                                                                                                                           
          
          ERROR: undefined method `getServers' for nil:NilClass
          
          Here is some help for this command:
          Get a region server group's information.
          
          Example:
          
            hbase> get_group 'default'
          

          and

          hbase(main):077:0* move_group_tables 'nonexisting'
          
          ERROR: undefined method `each' for nil:NilClass
          
          Here is some help for this command:
          Reassign tables from one group to another.
          
            hbase> move_group_tables 'dest',['table1','table2']
          

          and

          hbase(main):173:0* move_group_servers 'nonexisting'
          
          ERROR: undefined method `each' for nil:NilClass
          
          Here is some help for this command:
          Reassign a region server from one group to another.
          
            hbase> move_group_servers 'dest',['server1:port','server2:port']
          

          3. Group names should be restricted to alphanumeric only. This one is pretty easy, but important. This following caused the master to abort, and the master cannot restart after this point (without manually removing the rsgroup entry from the table which you cannot do without master). I had to nuke the hdfs and zk to start over.

          hbase(main):033:0> add_group 'a-/:*'
          
          ERROR: java.io.IOException: Failed to write to groupZNode
          	at org.apache.hadoop.hbase.group.GroupInfoManagerImpl.flushConfig(GroupInfoManagerImpl.java:419)
          	at org.apache.hadoop.hbase.group.GroupInfoManagerImpl.addGroup(GroupInfoManagerImpl.java:152)
          	at org.apache.hadoop.hbase.group.GroupAdminServer.addGroup(GroupAdminServer.java:298)
          	at org.apache.hadoop.hbase.group.GroupAdminEndpoint.addGroup(GroupAdminEndpoint.java:197)
          	at org.apache.hadoop.hbase.protobuf.generated.GroupAdminProtos$GroupAdminService.callMethod(GroupAdminProtos.java:11146)
          	at org.apache.hadoop.hbase.master.MasterRpcServices.execMasterService(MasterRpcServices.java:666)
          	at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:51121)
          	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
          	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
          	at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
          	at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
          	at java.lang.Thread.run(Thread.java:745)
          Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase-unsecure/groupInfo/a-/:*
          	at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
          	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
          	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
          	at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.createNonSequential(RecoverableZooKeeper.java:575)
          	at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.create(RecoverableZooKeeper.java:554)
          	at org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:1261)
          	at org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:1250)
          	at org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:1233)
          	at org.apache.hadoop.hbase.group.GroupInfoManagerImpl.flushConfig(GroupInfoManagerImpl.java:408)
          

          4. get_table_group and get_server_group shell commands do not work

          hbase(main):019:0* get_table_group 'nonexisting'
          
          ERROR: undefined local variable or method `s' for #<Hbase::GroupAdmin:0x64518270>
          
          Here is some help for this command:
          Get the group name the given table is a member of.
          
            hbase> get_table_group 'myTable'
          
           
          hbase(main):022:0* get_server_group 'server'
          
          ERROR: undefined local variable or method `s' for #<Hbase::GroupAdmin:0x64518270>
          
          Here is some help for this command:
          Get the group name the given region server is a member of.
          
            hbase> get_server_group 'server1:port1
          

          5. move_group_servers and move_group_tables arguments are listed as 1, although should be 2:

          hbase(main):033:0* move_group_servers 
          
          ERROR: wrong number of arguments (0 for 1)
          
          Here is some help for this command:
          Reassign a region server from one group to another.
          
            hbase> move_group_servers 'dest',['server1:port','server2:port']
          

          6. Adding a server without port throws error, but no explanation (this one is a minor, not that important).

          hbase(main):070:0> move_group_servers 'group2', ['os-enis-hbase-oct27-a-3.novalocal']  
          
          ERROR: 
          
          Here is some help for this command:
          Reassign a region server from one group to another.
          
            hbase> move_group_servers 'dest',['server1:port','server2:port']
          

          7. From all the above, it is clear that we need a unit test over the new shell commands.

          Other than these, the feature is working as expected. Defining groups, moving servers and tables work. Regions get reassigned according to their groups. Restarting the cluster keeps assignments, etc.

          Some more findings:
          Test 1:
          Killed the last regionserver of a group, all 15 regions are in FAILED_OPEN state.

          • restarted the master, regions still in FAILED_OPEN state (which is expected)
          • Added a new server to the group which had no remaining servers, regions still in FAILED_OPEN state (this is probably due to how assignment works, we give up after 10 retries and wait for manual assignment or master restart)
          • Started the region server that was killed before, still in FAILED_OPEN
          • Master restart reassigned these regions.

          Test 2:
          Tried to move all servers to a single group. Correctly handles last server in the default group by not allowing it to change.

          Test 3:
          Killed the last server in the default group, while all system tables are in the default group (and hence in that server).
          -> hbase:meta was always in PENDING_OPEN in bogus server localhost,1,1.
          -> Upon restarting the killed server, meta and other tables in the default group (including rsgroup table) got reassigned.
          As a side note, having not enough servers in the group that has the meta or rsgroup table seems like a very good way of shoothing yourself in the foot. However, as discussed before this maybe needed for strong isolation.

          • Add non-existing server to the group. Is not allowed.
          • Checked JMX
          • Adding group information for tables and regionserver to the master UI would be helpful. We can leave this to a follow up.
          • Obviously there should be a follow up to add at least some basic documentation on how to enable and configure and use RS groups in the book.
          Show
          enis Enis Soztutar added a comment - Finally got around testing the v15 patch on 1.1 code base with a 7 node cluster. Here are my test notes. Nothing too concerning, but we have to address some of these in the patch. This is the configuration to add to enable groups: <property> <name>hbase.coprocessor.master.classes</name> <value>org.apache.hadoop.hbase.group.GroupAdminEndpoint</value> </property> <property> <name>hbase.master.loadbalancer.class</name> <value>org.apache.hadoop.hbase.group.GroupBasedLoadBalancer</value> </property> 1. Need to add this diff, so that new PB files get compiled with -Pcompile-protobuf command: diff --git hbase-protocol/pom.xml hbase-protocol/pom.xml index 8034576..d352373 100644 --- hbase-protocol/pom.xml +++ hbase-protocol/pom.xml @@ -180,6 +180,8 @@ <include>ErrorHandling.proto</include> <include>Filter.proto</include> <include>FS.proto</include> + <include>Group.proto</include> + <include>GroupAdmin.proto</include> <include>HBase.proto</include> <include>HFile.proto</include> <include>LoadBalancer.proto</include> 2. NPE in group shell commands with nonexisting groups: hbase(main):015:0* balance_group 'nonexisting' ERROR: java.io.IOException at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2156) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang. Thread .run( Thread .java:745) Caused by: java.lang.NullPointerException at org.apache.hadoop.hbase.group.GroupAdminServer.groupGetRegionsInTransition(GroupAdminServer.java:412) at org.apache.hadoop.hbase.group.GroupAdminServer.balanceGroup(GroupAdminServer.java:348) at org.apache.hadoop.hbase.group.GroupAdminEndpoint.balanceGroup(GroupAdminEndpoint.java:229) at org.apache.hadoop.hbase.protobuf.generated.GroupAdminProtos$GroupAdminService.callMethod(GroupAdminProtos.java:11156) at org.apache.hadoop.hbase.master.MasterRpcServices.execMasterService(MasterRpcServices.java:666) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:51121) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114) and hbase(main):030:0> get_group 'nonexisting' GROUP INFORMATION Servers: ERROR: undefined method `getServers' for nil:NilClass Here is some help for this command: Get a region server group's information. Example: hbase> get_group ' default ' and hbase(main):077:0* move_group_tables 'nonexisting' ERROR: undefined method `each' for nil:NilClass Here is some help for this command: Reassign tables from one group to another. hbase> move_group_tables 'dest',['table1','table2'] and hbase(main):173:0* move_group_servers 'nonexisting' ERROR: undefined method `each' for nil:NilClass Here is some help for this command: Reassign a region server from one group to another. hbase> move_group_servers 'dest',['server1:port','server2:port'] 3. Group names should be restricted to alphanumeric only. This one is pretty easy, but important. This following caused the master to abort, and the master cannot restart after this point (without manually removing the rsgroup entry from the table which you cannot do without master). I had to nuke the hdfs and zk to start over. hbase(main):033:0> add_group 'a-/:*' ERROR: java.io.IOException: Failed to write to groupZNode at org.apache.hadoop.hbase.group.GroupInfoManagerImpl.flushConfig(GroupInfoManagerImpl.java:419) at org.apache.hadoop.hbase.group.GroupInfoManagerImpl.addGroup(GroupInfoManagerImpl.java:152) at org.apache.hadoop.hbase.group.GroupAdminServer.addGroup(GroupAdminServer.java:298) at org.apache.hadoop.hbase.group.GroupAdminEndpoint.addGroup(GroupAdminEndpoint.java:197) at org.apache.hadoop.hbase.protobuf.generated.GroupAdminProtos$GroupAdminService.callMethod(GroupAdminProtos.java:11146) at org.apache.hadoop.hbase.master.MasterRpcServices.execMasterService(MasterRpcServices.java:666) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:51121) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang. Thread .run( Thread .java:745) Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase-unsecure/groupInfo/a-/:* at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.createNonSequential(RecoverableZooKeeper.java:575) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.create(RecoverableZooKeeper.java:554) at org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:1261) at org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:1250) at org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:1233) at org.apache.hadoop.hbase.group.GroupInfoManagerImpl.flushConfig(GroupInfoManagerImpl.java:408) 4. get_table_group and get_server_group shell commands do not work hbase(main):019:0* get_table_group 'nonexisting' ERROR: undefined local variable or method `s' for #<Hbase::GroupAdmin:0x64518270> Here is some help for this command: Get the group name the given table is a member of. hbase> get_table_group 'myTable' hbase(main):022:0* get_server_group 'server' ERROR: undefined local variable or method `s' for #<Hbase::GroupAdmin:0x64518270> Here is some help for this command: Get the group name the given region server is a member of. hbase> get_server_group 'server1:port1 5. move_group_servers and move_group_tables arguments are listed as 1, although should be 2: hbase(main):033:0* move_group_servers ERROR: wrong number of arguments (0 for 1) Here is some help for this command: Reassign a region server from one group to another. hbase> move_group_servers 'dest',['server1:port','server2:port'] 6. Adding a server without port throws error, but no explanation (this one is a minor, not that important). hbase(main):070:0> move_group_servers 'group2', ['os-enis-hbase-oct27-a-3.novalocal'] ERROR: Here is some help for this command: Reassign a region server from one group to another. hbase> move_group_servers 'dest',['server1:port','server2:port'] 7. From all the above, it is clear that we need a unit test over the new shell commands. Other than these, the feature is working as expected. Defining groups, moving servers and tables work. Regions get reassigned according to their groups. Restarting the cluster keeps assignments, etc. Some more findings: Test 1: Killed the last regionserver of a group, all 15 regions are in FAILED_OPEN state. restarted the master, regions still in FAILED_OPEN state (which is expected) Added a new server to the group which had no remaining servers, regions still in FAILED_OPEN state (this is probably due to how assignment works, we give up after 10 retries and wait for manual assignment or master restart) Started the region server that was killed before, still in FAILED_OPEN Master restart reassigned these regions. Test 2: Tried to move all servers to a single group. Correctly handles last server in the default group by not allowing it to change. Test 3: Killed the last server in the default group, while all system tables are in the default group (and hence in that server). -> hbase:meta was always in PENDING_OPEN in bogus server localhost,1,1. -> Upon restarting the killed server, meta and other tables in the default group (including rsgroup table) got reassigned. As a side note, having not enough servers in the group that has the meta or rsgroup table seems like a very good way of shoothing yourself in the foot. However, as discussed before this maybe needed for strong isolation. Add non-existing server to the group. Is not allowed. Checked JMX Adding group information for tables and regionserver to the master UI would be helpful. We can leave this to a follow up. Obviously there should be a follow up to add at least some basic documentation on how to enable and configure and use RS groups in the book.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12768882/hbase-6721-v15-branch-1.1.patch
          against branch-1.1 branch at commit 496d20cfca5a30bc72a29e4ef893424964f9fa91.
          ATTACHMENT ID: 12768882

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 33 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/16242//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12768882/hbase-6721-v15-branch-1.1.patch against branch-1.1 branch at commit 496d20cfca5a30bc72a29e4ef893424964f9fa91. ATTACHMENT ID: 12768882 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 33 new or modified tests. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/16242//console This message is automatically generated.
          Hide
          enis Enis Soztutar added a comment -

          I backported the current v15 patch for testing it with 1.1 code base. Just parking it here in case somebody else needs it. I'll get back to the review soon.

          Show
          enis Enis Soztutar added a comment - I backported the current v15 patch for testing it with 1.1 code base. Just parking it here in case somebody else needs it. I'll get back to the review soon.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12764454/HBASE-6721_15.patch
          against master branch at commit 24370547c500df0026a71944d8be88cd5b51b23e.
          ATTACHMENT ID: 12764454

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 33 new or modified tests.

          +1 hadoop versions. The patch compiles with all supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0 2.7.1)

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 protoc. The applied patch does not increase the total number of protoc compiler warnings.

          -1 javadoc. The javadoc tool appears to have generated 3 warning messages.

          -1 checkstyle. The applied patch generated 1811 checkstyle errors (more than the master's current 1781 errors).

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 lineLengths. The patch introduces the following lines longer than 100:
          + * <code>rpc GetGroupInfo(.hbase.pb.GetGroupInfoRequest) returns (.hbase.pb.GetGroupInfoResponse);</code>
          + * <code>rpc GetGroupInfoOfTable(.hbase.pb.GetGroupInfoOfTableRequest) returns (.hbase.pb.GetGroupInfoOfTableResponse);</code>
          + * <code>rpc GetGroupInfoOfServer(.hbase.pb.GetGroupInfoOfServerRequest) returns (.hbase.pb.GetGroupInfoOfServerResponse);</code>
          + * <code>rpc MoveServers(.hbase.pb.MoveServersRequest) returns (.hbase.pb.MoveServersResponse);</code>
          + * <code>rpc MoveTables(.hbase.pb.MoveTablesRequest) returns (.hbase.pb.MoveTablesResponse);</code>
          + * <code>rpc RemoveGroup(.hbase.pb.RemoveGroupRequest) returns (.hbase.pb.RemoveGroupResponse);</code>
          + * <code>rpc BalanceGroup(.hbase.pb.BalanceGroupRequest) returns (.hbase.pb.BalanceGroupResponse);</code>
          + * <code>rpc ListGroupInfos(.hbase.pb.ListGroupInfosRequest) returns (.hbase.pb.ListGroupInfosResponse);</code>
          + * <code>rpc GetGroupInfo(.hbase.pb.GetGroupInfoRequest) returns (.hbase.pb.GetGroupInfoResponse);</code>
          + * <code>rpc GetGroupInfoOfTable(.hbase.pb.GetGroupInfoOfTableRequest) returns (.hbase.pb.GetGroupInfoOfTableResponse);</code>

          +1 site. The mvn post-site goal succeeds with this patch.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.client.TestAsyncProcess

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/15832//testReport/
          Release Findbugs (version 2.0.3) warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/15832//artifact/patchprocess/newFindbugsWarnings.html
          Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/15832//artifact/patchprocess/checkstyle-aggregate.html

          Javadoc warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/15832//artifact/patchprocess/patchJavadocWarnings.txt
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/15832//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12764454/HBASE-6721_15.patch against master branch at commit 24370547c500df0026a71944d8be88cd5b51b23e. ATTACHMENT ID: 12764454 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 33 new or modified tests. +1 hadoop versions . The patch compiles with all supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0 2.7.1) +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 protoc . The applied patch does not increase the total number of protoc compiler warnings. -1 javadoc . The javadoc tool appears to have generated 3 warning messages. -1 checkstyle . The applied patch generated 1811 checkstyle errors (more than the master's current 1781 errors). +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 lineLengths . The patch introduces the following lines longer than 100: + * <code>rpc GetGroupInfo(.hbase.pb.GetGroupInfoRequest) returns (.hbase.pb.GetGroupInfoResponse);</code> + * <code>rpc GetGroupInfoOfTable(.hbase.pb.GetGroupInfoOfTableRequest) returns (.hbase.pb.GetGroupInfoOfTableResponse);</code> + * <code>rpc GetGroupInfoOfServer(.hbase.pb.GetGroupInfoOfServerRequest) returns (.hbase.pb.GetGroupInfoOfServerResponse);</code> + * <code>rpc MoveServers(.hbase.pb.MoveServersRequest) returns (.hbase.pb.MoveServersResponse);</code> + * <code>rpc MoveTables(.hbase.pb.MoveTablesRequest) returns (.hbase.pb.MoveTablesResponse);</code> + * <code>rpc RemoveGroup(.hbase.pb.RemoveGroupRequest) returns (.hbase.pb.RemoveGroupResponse);</code> + * <code>rpc BalanceGroup(.hbase.pb.BalanceGroupRequest) returns (.hbase.pb.BalanceGroupResponse);</code> + * <code>rpc ListGroupInfos(.hbase.pb.ListGroupInfosRequest) returns (.hbase.pb.ListGroupInfosResponse);</code> + * <code>rpc GetGroupInfo(.hbase.pb.GetGroupInfoRequest) returns (.hbase.pb.GetGroupInfoResponse);</code> + * <code>rpc GetGroupInfoOfTable(.hbase.pb.GetGroupInfoOfTableRequest) returns (.hbase.pb.GetGroupInfoOfTableResponse);</code> +1 site . The mvn post-site goal succeeds with this patch. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.client.TestAsyncProcess Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/15832//testReport/ Release Findbugs (version 2.0.3) warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/15832//artifact/patchprocess/newFindbugsWarnings.html Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/15832//artifact/patchprocess/checkstyle-aggregate.html Javadoc warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/15832//artifact/patchprocess/patchJavadocWarnings.txt Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/15832//console This message is automatically generated.
          Hide
          toffer Francis Liu added a comment -

          Addressed comments on RB.

          Andrew Purtell let me know if the changes I made for security is what you expected. I pretty much just ported the security/cp stuff from the non-cp patch.

          Show
          toffer Francis Liu added a comment - Addressed comments on RB. Andrew Purtell let me know if the changes I made for security is what you expected. I pretty much just ported the security/cp stuff from the non-cp patch.
          Hide
          apurtell Andrew Purtell added a comment - - edited

          Andrew Purtell The new patch is based off of current master. Should we just replace the current branch with a new branch to go with the new patch?

          I would say so if you want it.

          Also was wondering since this is now cp-based. Do we still need a branch?

          Up to you. We can rebase or delete, either way let me know.

          Some issues with the current security integration. Coprocessors can't call into the internals of other coprocessors. I understand why this was done, but we can't have it. Coprocessors calling into the internals of other coprocessors, this is a non-negotiable point for the sake of sanity in maintenance of separate optional extensions. It's a catch-22 imposed on this change by the requirement it be a coprocessor only implementation.

          What I would suggest is introduce into the MasterObserver API hooks for the group admin APIs. Let the implementation of the group admin APIs and the authoritative security decisions both be separate mix-ins provided by different coprocessors. There needs to be common plumbing for the two. That belongs in MasterObserver. The plumbing could look like:

          • MasterObserver support for pre/post group admin API action hooks
          • In GroupAdminEndpoint, get the coprocessor host with getMasterCoprocessorHost()
          • Invoke the public (technically, LimitedPrivate(COPROC)) APIs for pre/post group admin API actions.
          • AccessController implements the new MasterObserver APIs to provide security for the group admin APIs.

          This is much more in spirit with current interfaces and audience scoping. It decouples GroupAdminEndpoint from AccessController. (If the AC is not installed, no harm, no NPEs, no security checking (by intention), it's all good.) It also addresses concerns about zero impact in the default case. Those upcalls will never be made unless the GroupAdminEndpoint is installed.

          Show
          apurtell Andrew Purtell added a comment - - edited Andrew Purtell The new patch is based off of current master. Should we just replace the current branch with a new branch to go with the new patch? I would say so if you want it. Also was wondering since this is now cp-based. Do we still need a branch? Up to you. We can rebase or delete, either way let me know. Some issues with the current security integration. Coprocessors can't call into the internals of other coprocessors. I understand why this was done, but we can't have it. Coprocessors calling into the internals of other coprocessors, this is a non-negotiable point for the sake of sanity in maintenance of separate optional extensions. It's a catch-22 imposed on this change by the requirement it be a coprocessor only implementation. What I would suggest is introduce into the MasterObserver API hooks for the group admin APIs. Let the implementation of the group admin APIs and the authoritative security decisions both be separate mix-ins provided by different coprocessors. There needs to be common plumbing for the two. That belongs in MasterObserver. The plumbing could look like: MasterObserver support for pre/post group admin API action hooks In GroupAdminEndpoint, get the coprocessor host with getMasterCoprocessorHost() Invoke the public (technically, LimitedPrivate(COPROC)) APIs for pre/post group admin API actions. AccessController implements the new MasterObserver APIs to provide security for the group admin APIs. This is much more in spirit with current interfaces and audience scoping. It decouples GroupAdminEndpoint from AccessController. (If the AC is not installed, no harm, no NPEs, no security checking (by intention), it's all good.) It also addresses concerns about zero impact in the default case. Those upcalls will never be made unless the GroupAdminEndpoint is installed.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12756192/HBASE-6721_14.patch
          against master branch at commit d2e338181800ae3cef55ddca491901b65259dc7f.
          ATTACHMENT ID: 12756192

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 27 new or modified tests.

          +1 hadoop versions. The patch compiles with all supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0 2.7.1)

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 protoc. The applied patch does not increase the total number of protoc compiler warnings.

          -1 javadoc. The javadoc tool appears to have generated 6 warning messages.

          -1 checkstyle. The applied patch generated 1859 checkstyle errors (more than the master's current 1835 errors).

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 lineLengths. The patch introduces the following lines longer than 100:
          + * <code>rpc GetGroupInfo(.hbase.pb.GetGroupInfoRequest) returns (.hbase.pb.GetGroupInfoResponse);</code>
          + * <code>rpc GetGroupInfoOfTable(.hbase.pb.GetGroupInfoOfTableRequest) returns (.hbase.pb.GetGroupInfoOfTableResponse);</code>
          + * <code>rpc GetGroupInfoOfServer(.hbase.pb.GetGroupInfoOfServerRequest) returns (.hbase.pb.GetGroupInfoOfServerResponse);</code>
          + * <code>rpc MoveServers(.hbase.pb.MoveServersRequest) returns (.hbase.pb.MoveServersResponse);</code>
          + * <code>rpc MoveTables(.hbase.pb.MoveTablesRequest) returns (.hbase.pb.MoveTablesResponse);</code>
          + * <code>rpc RemoveGroup(.hbase.pb.RemoveGroupRequest) returns (.hbase.pb.RemoveGroupResponse);</code>
          + * <code>rpc BalanceGroup(.hbase.pb.BalanceGroupRequest) returns (.hbase.pb.BalanceGroupResponse);</code>
          + * <code>rpc ListGroupInfos(.hbase.pb.ListGroupInfosRequest) returns (.hbase.pb.ListGroupInfosResponse);</code>
          + * <code>rpc GetGroupInfo(.hbase.pb.GetGroupInfoRequest) returns (.hbase.pb.GetGroupInfoResponse);</code>
          + * <code>rpc GetGroupInfoOfTable(.hbase.pb.GetGroupInfoOfTableRequest) returns (.hbase.pb.GetGroupInfoOfTableResponse);</code>

          +1 site. The mvn post-site goal succeeds with this patch.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.client.TestAsyncProcess

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/15618//testReport/
          Release Findbugs (version 2.0.3) warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/15618//artifact/patchprocess/newFindbugsWarnings.html
          Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/15618//artifact/patchprocess/checkstyle-aggregate.html

          Javadoc warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/15618//artifact/patchprocess/patchJavadocWarnings.txt
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/15618//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12756192/HBASE-6721_14.patch against master branch at commit d2e338181800ae3cef55ddca491901b65259dc7f. ATTACHMENT ID: 12756192 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 27 new or modified tests. +1 hadoop versions . The patch compiles with all supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0 2.7.1) +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 protoc . The applied patch does not increase the total number of protoc compiler warnings. -1 javadoc . The javadoc tool appears to have generated 6 warning messages. -1 checkstyle . The applied patch generated 1859 checkstyle errors (more than the master's current 1835 errors). +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 lineLengths . The patch introduces the following lines longer than 100: + * <code>rpc GetGroupInfo(.hbase.pb.GetGroupInfoRequest) returns (.hbase.pb.GetGroupInfoResponse);</code> + * <code>rpc GetGroupInfoOfTable(.hbase.pb.GetGroupInfoOfTableRequest) returns (.hbase.pb.GetGroupInfoOfTableResponse);</code> + * <code>rpc GetGroupInfoOfServer(.hbase.pb.GetGroupInfoOfServerRequest) returns (.hbase.pb.GetGroupInfoOfServerResponse);</code> + * <code>rpc MoveServers(.hbase.pb.MoveServersRequest) returns (.hbase.pb.MoveServersResponse);</code> + * <code>rpc MoveTables(.hbase.pb.MoveTablesRequest) returns (.hbase.pb.MoveTablesResponse);</code> + * <code>rpc RemoveGroup(.hbase.pb.RemoveGroupRequest) returns (.hbase.pb.RemoveGroupResponse);</code> + * <code>rpc BalanceGroup(.hbase.pb.BalanceGroupRequest) returns (.hbase.pb.BalanceGroupResponse);</code> + * <code>rpc ListGroupInfos(.hbase.pb.ListGroupInfosRequest) returns (.hbase.pb.ListGroupInfosResponse);</code> + * <code>rpc GetGroupInfo(.hbase.pb.GetGroupInfoRequest) returns (.hbase.pb.GetGroupInfoResponse);</code> + * <code>rpc GetGroupInfoOfTable(.hbase.pb.GetGroupInfoOfTableRequest) returns (.hbase.pb.GetGroupInfoOfTableResponse);</code> +1 site . The mvn post-site goal succeeds with this patch. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.client.TestAsyncProcess Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/15618//testReport/ Release Findbugs (version 2.0.3) warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/15618//artifact/patchprocess/newFindbugsWarnings.html Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/15618//artifact/patchprocess/checkstyle-aggregate.html Javadoc warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/15618//artifact/patchprocess/patchJavadocWarnings.txt Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/15618//console This message is automatically generated.
          Hide
          toffer Francis Liu added a comment -

          Addressed comments on RB.

          Show
          toffer Francis Liu added a comment - Addressed comments on RB.
          Hide
          eclark Elliott Clark added a comment -

          Commenting on RB now.

          Show
          eclark Elliott Clark added a comment - Commenting on RB now.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12754964/HBASE-6721_13.patch
          against master branch at commit 27d3ab43efeabe2a0e1173858b6994b17b5c355b.
          ATTACHMENT ID: 12754964

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 27 new or modified tests.

          +1 hadoop versions. The patch compiles with all supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0 2.7.1)

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 protoc. The applied patch does not increase the total number of protoc compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 checkstyle. The applied patch does not increase the total number of checkstyle errors

          -1 findbugs. The patch appears to cause Findbugs (version 2.0.3) to fail.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 lineLengths. The patch introduces the following lines longer than 100:
          + * <code>rpc GetGroupInfo(.hbase.pb.GetGroupInfoRequest) returns (.hbase.pb.GetGroupInfoResponse);</code>
          + * <code>rpc GetGroupInfoOfTable(.hbase.pb.GetGroupInfoOfTableRequest) returns (.hbase.pb.GetGroupInfoOfTableResponse);</code>
          + * <code>rpc GetGroupInfoOfServer(.hbase.pb.GetGroupInfoOfServerRequest) returns (.hbase.pb.GetGroupInfoOfServerResponse);</code>
          + * <code>rpc MoveServers(.hbase.pb.MoveServersRequest) returns (.hbase.pb.MoveServersResponse);</code>
          + * <code>rpc MoveTables(.hbase.pb.MoveTablesRequest) returns (.hbase.pb.MoveTablesResponse);</code>
          + * <code>rpc RemoveGroup(.hbase.pb.RemoveGroupRequest) returns (.hbase.pb.RemoveGroupResponse);</code>
          + * <code>rpc BalanceGroup(.hbase.pb.BalanceGroupRequest) returns (.hbase.pb.BalanceGroupResponse);</code>
          + * <code>rpc ListGroupInfos(.hbase.pb.ListGroupInfosRequest) returns (.hbase.pb.ListGroupInfosResponse);</code>
          + * <code>rpc GetGroupInfo(.hbase.pb.GetGroupInfoRequest) returns (.hbase.pb.GetGroupInfoResponse);</code>
          + * <code>rpc GetGroupInfoOfTable(.hbase.pb.GetGroupInfoOfTableRequest) returns (.hbase.pb.GetGroupInfoOfTableResponse);</code>

          -1 site. The patch appears to cause mvn post-site goal to fail.

          -1 core tests. The patch failed these unit tests:

          -1 core zombie tests. There are 1 zombie test(s):

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/15506//testReport/
          Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/15506//artifact/patchprocess/checkstyle-aggregate.html

          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/15506//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12754964/HBASE-6721_13.patch against master branch at commit 27d3ab43efeabe2a0e1173858b6994b17b5c355b. ATTACHMENT ID: 12754964 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 27 new or modified tests. +1 hadoop versions . The patch compiles with all supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0 2.7.1) +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 protoc . The applied patch does not increase the total number of protoc compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 checkstyle . The applied patch does not increase the total number of checkstyle errors -1 findbugs . The patch appears to cause Findbugs (version 2.0.3) to fail. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 lineLengths . The patch introduces the following lines longer than 100: + * <code>rpc GetGroupInfo(.hbase.pb.GetGroupInfoRequest) returns (.hbase.pb.GetGroupInfoResponse);</code> + * <code>rpc GetGroupInfoOfTable(.hbase.pb.GetGroupInfoOfTableRequest) returns (.hbase.pb.GetGroupInfoOfTableResponse);</code> + * <code>rpc GetGroupInfoOfServer(.hbase.pb.GetGroupInfoOfServerRequest) returns (.hbase.pb.GetGroupInfoOfServerResponse);</code> + * <code>rpc MoveServers(.hbase.pb.MoveServersRequest) returns (.hbase.pb.MoveServersResponse);</code> + * <code>rpc MoveTables(.hbase.pb.MoveTablesRequest) returns (.hbase.pb.MoveTablesResponse);</code> + * <code>rpc RemoveGroup(.hbase.pb.RemoveGroupRequest) returns (.hbase.pb.RemoveGroupResponse);</code> + * <code>rpc BalanceGroup(.hbase.pb.BalanceGroupRequest) returns (.hbase.pb.BalanceGroupResponse);</code> + * <code>rpc ListGroupInfos(.hbase.pb.ListGroupInfosRequest) returns (.hbase.pb.ListGroupInfosResponse);</code> + * <code>rpc GetGroupInfo(.hbase.pb.GetGroupInfoRequest) returns (.hbase.pb.GetGroupInfoResponse);</code> + * <code>rpc GetGroupInfoOfTable(.hbase.pb.GetGroupInfoOfTableRequest) returns (.hbase.pb.GetGroupInfoOfTableResponse);</code> -1 site . The patch appears to cause mvn post-site goal to fail. -1 core tests . The patch failed these unit tests: -1 core zombie tests . There are 1 zombie test(s): Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/15506//testReport/ Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/15506//artifact/patchprocess/checkstyle-aggregate.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/15506//console This message is automatically generated.
          Hide
          toffer Francis Liu added a comment -

          Latest patch attached. The patch makes the implementation cp-based.

          To avoid confusion I've created a new RB entry:

          https://reviews.apache.org/r/38224/

          Andrew Purtell The new patch is based off of current master. Should we just replace the current branch with a new branch to go with the new patch? Also was wondering since this is now cp-based. Do we still need a branch?

          Elliott Clark I believe I've addressed most of your concerns. The only thing I missed was "Load meta's hri from disk on move" can you elaborate more one what needs to be done on RB?

          Show
          toffer Francis Liu added a comment - Latest patch attached. The patch makes the implementation cp-based. To avoid confusion I've created a new RB entry: https://reviews.apache.org/r/38224/ Andrew Purtell The new patch is based off of current master. Should we just replace the current branch with a new branch to go with the new patch? Also was wondering since this is now cp-based. Do we still need a branch? Elliott Clark I believe I've addressed most of your concerns. The only thing I missed was "Load meta's hri from disk on move" can you elaborate more one what needs to be done on RB?
          Hide
          enis Enis Soztutar added a comment -

          Agreed, that let's keep it simple for now unless needed.

          Show
          enis Enis Soztutar added a comment - Agreed, that let's keep it simple for now unless needed.
          Hide
          apurtell Andrew Purtell added a comment -

          But it will be a coprocessor having its own co-processors which seems is not needed at the moment.

          Yeah, let's avoid that until/unless we really need it

          Show
          apurtell Andrew Purtell added a comment - But it will be a coprocessor having its own co-processors which seems is not needed at the moment. Yeah, let's avoid that until/unless we really need it
          Hide
          toffer Francis Liu added a comment -

          Enis Soztutar I can do that too and have security make use of it. Should be just as much effort as embedding security directly as I already have the RS group cp hooks in place as part of the current patch?

          Show
          toffer Francis Liu added a comment - Enis Soztutar I can do that too and have security make use of it. Should be just as much effort as embedding security directly as I already have the RS group cp hooks in place as part of the current patch?
          Hide
          enis Enis Soztutar added a comment -

          Great.
          We can in theory define another Observer type (RSGroupObserver, etc) which will be the endpoint that coprocessors might implement for example for learning about RS groups operations. But it will be a coprocessor having its own co-processors which seems is not needed at the moment.

          Show
          enis Enis Soztutar added a comment - Great. We can in theory define another Observer type (RSGroupObserver, etc) which will be the endpoint that coprocessors might implement for example for learning about RS groups operations. But it will be a coprocessor having its own co-processors which seems is not needed at the moment.
          Hide
          eclark Elliott Clark added a comment -

          That works great for me and would address the concerns I had.

          Show
          eclark Elliott Clark added a comment - That works great for me and would address the concerns I had.
          Hide
          toffer Francis Liu added a comment -

          In the interest of moving this feature forward I propose we change the current patch to use a coprocessor-based implementation. The broad strokes of the implementation are:

          1. Group APIs will move to a coprocessor endpoint
          2. Other hooks into core code will utilize coprocessor observers and possibly add more cp hooks in core as needed.
          3. Security checks will be embedded directly into the endpoint, which will be activated if security is enabled.
          4. Add new api/semantic in LoadBalancer so that it can tell callees when regions should not be assigned (ie bogus server name).
          5. CLI will remain as-is (same as security)

          Elliott Clark does this work for you?

          Show
          toffer Francis Liu added a comment - In the interest of moving this feature forward I propose we change the current patch to use a coprocessor-based implementation. The broad strokes of the implementation are: 1. Group APIs will move to a coprocessor endpoint 2. Other hooks into core code will utilize coprocessor observers and possibly add more cp hooks in core as needed. 3. Security checks will be embedded directly into the endpoint, which will be activated if security is enabled. 4. Add new api/semantic in LoadBalancer so that it can tell callees when regions should not be assigned (ie bogus server name). 5. CLI will remain as-is (same as security) Elliott Clark does this work for you?
          Hide
          apurtell Andrew Purtell added a comment -

          Yeah and I have evaluated it and used my knowledge and judgement and cast my vote on this patch and feature in accordance with the Apache Foundation rules (http://www.apache.org/foundation/voting.html#votes-on-code-modification) . I have added my technical reasons

          Let me collect them here:

          • Don't break ipv6.
          • Remove Copy Paste code.
          • Add correct headers.
          • Load meta's hri from disk on move.
          • No added complexity on the default path

          I'd like to add:

          • Ops impact should be totally optional: configurable, default off. (This is related to the last point above.)

          If I've missed any items please let me know.

          Show
          apurtell Andrew Purtell added a comment - Yeah and I have evaluated it and used my knowledge and judgement and cast my vote on this patch and feature in accordance with the Apache Foundation rules ( http://www.apache.org/foundation/voting.html#votes-on-code-modification ) . I have added my technical reasons Let me collect them here: Don't break ipv6. Remove Copy Paste code. Add correct headers. Load meta's hri from disk on move. No added complexity on the default path I'd like to add: Ops impact should be totally optional: configurable, default off. (This is related to the last point above.) If I've missed any items please let me know.
          Hide
          apurtell Andrew Purtell added a comment -

          No added complexity on the default use case.

          With the group assignment logic changes behind a default off conditional that criteria is met here too.

          Show
          apurtell Andrew Purtell added a comment - No added complexity on the default use case. With the group assignment logic changes behind a default off conditional that criteria is met here too.
          Hide
          eclark Elliott Clark added a comment -

          Maybe you can elaborate on this point?

          Sure let me write up something more on that point.

          Let's not let the perfect be the enemy of the good.

          I'm not asking for perfect. I'm asking not to take a half step back for most users so that one user can merge this feature and take a single step forward.

          Meanwhile we have a patch on deck and we need to be evaluating it and its contributor's concerns on their merit.

          Yeah and I have evaluated it and used my knowledge and judgement and cast my vote on this patch and feature in accordance with the Apache Foundation rules (http://www.apache.org/foundation/voting.html#votes-on-code-modification) . I have added my technical reasons. I have even outlined what I would need to vote a different way.

          the 0.90 master rewrite (remember that?)

          Yeah that has since proved to be the wrong way to go. It put way too much in ZK and now we've spent years un-doing it. We would have been better served with asking for parts to be pluggable and not on by default.

          or the memcache-based block cache carried

          That followed the exact same bar that I'm requesting here. No added complexity on the default use case. I've even moved it into a different module so that it will be exactly what I'm asking for here.

          Show
          eclark Elliott Clark added a comment - Maybe you can elaborate on this point? Sure let me write up something more on that point. Let's not let the perfect be the enemy of the good. I'm not asking for perfect. I'm asking not to take a half step back for most users so that one user can merge this feature and take a single step forward. Meanwhile we have a patch on deck and we need to be evaluating it and its contributor's concerns on their merit. Yeah and I have evaluated it and used my knowledge and judgement and cast my vote on this patch and feature in accordance with the Apache Foundation rules ( http://www.apache.org/foundation/voting.html#votes-on-code-modification ) . I have added my technical reasons. I have even outlined what I would need to vote a different way. the 0.90 master rewrite (remember that?) Yeah that has since proved to be the wrong way to go. It put way too much in ZK and now we've spent years un-doing it. We would have been better served with asking for parts to be pluggable and not on by default. or the memcache-based block cache carried That followed the exact same bar that I'm requesting here. No added complexity on the default use case. I've even moved it into a different module so that it will be exactly what I'm asking for here.
          Hide
          apurtell Andrew Purtell added a comment -

          Let's not let the perfect be the enemy of the good. A "proper" multi-layer admission control change isn't on the table, it isn't on anyone's roadmap, it isn't even something proposed on a JIRA and/or in a design document. Even if we have a proposal for HBase this will certainly be considered imperfect and incomplete by some without 100% agreement and a plan at the HDFS level, and getting that is as likely as finding a unicorn wandering around downtown SF. (Ok... maybe a horse dressed to look like a unicorn could be a thing...) Meanwhile we have a patch on deck and we need to be evaluating it and its contributor's concerns on their merit.

          This is something that one of our esteemed users runs in production, is persistent about getting in and responsive to feedback, and both of those things in my opinion should carry a lot of weight. The same kind of weight that previously proposed changes like HFileV2 or the 0.90 master rewrite (remember that?), or the memcache-based block cache carried, or pending IPv6 related changes.

          That said, I don't think it's ready to be merged into master. We have it up in a feature branch. Let's continue that, address concerns, make sure it's totally optional for those who don't want it, measure its impact.

          Show
          apurtell Andrew Purtell added a comment - Let's not let the perfect be the enemy of the good. A "proper" multi-layer admission control change isn't on the table, it isn't on anyone's roadmap, it isn't even something proposed on a JIRA and/or in a design document. Even if we have a proposal for HBase this will certainly be considered imperfect and incomplete by some without 100% agreement and a plan at the HDFS level, and getting that is as likely as finding a unicorn wandering around downtown SF. (Ok... maybe a horse dressed to look like a unicorn could be a thing...) Meanwhile we have a patch on deck and we need to be evaluating it and its contributor's concerns on their merit. This is something that one of our esteemed users runs in production, is persistent about getting in and responsive to feedback, and both of those things in my opinion should carry a lot of weight. The same kind of weight that previously proposed changes like HFileV2 or the 0.90 master rewrite (remember that?), or the memcache-based block cache carried, or pending IPv6 related changes. That said, I don't think it's ready to be merged into master. We have it up in a feature branch. Let's continue that, address concerns, make sure it's totally optional for those who don't want it, measure its impact.
          Hide
          ndimiduk Nick Dimiduk added a comment -

          This is not only a multi-tenancy solution but an isolation solution.

          Isolation is the worse solution for getting multi-tenancy.

          Maybe you can elaborate on this point? Seems we need to quarantine users from each other, whether that's physically as per this patch or via imposed resource controls within a single process. Either way, "quarantine" is the same as "isolation"; we're isolating users from each other to achieve fairness of service delivery in a multi-tenant environment.

          Show
          ndimiduk Nick Dimiduk added a comment - This is not only a multi-tenancy solution but an isolation solution. Isolation is the worse solution for getting multi-tenancy. Maybe you can elaborate on this point? Seems we need to quarantine users from each other, whether that's physically as per this patch or via imposed resource controls within a single process. Either way, "quarantine" is the same as "isolation"; we're isolating users from each other to achieve fairness of service delivery in a multi-tenant environment.
          Hide
          eclark Elliott Clark added a comment -

          IMHO this is a huge oversimplification.

          All of the bad parts are the same. You still have to config it. You still have really weird behavior on cluster instability.

          Right now there's only group and it can have regionservers and tables as members.

          The patch has a group attached to namespaces, and also provides the ability to move tables to different groups.

          IMHO this will help a lot of users.

          Every user that's running this that doesn't have 10K machines is making a huge mistake. And we as the HBase developers should provide the better solution not the easiest solution. Not the one that works for one company.

          This is not only a multi-tenancy solution but an isolation solution.

          Isolation is the worse solution for getting multi-tenancy.

          Show
          eclark Elliott Clark added a comment - IMHO this is a huge oversimplification. All of the bad parts are the same. You still have to config it. You still have really weird behavior on cluster instability. Right now there's only group and it can have regionservers and tables as members. The patch has a group attached to namespaces, and also provides the ability to move tables to different groups. IMHO this will help a lot of users. Every user that's running this that doesn't have 10K machines is making a huge mistake. And we as the HBase developers should provide the better solution not the easiest solution. Not the one that works for one company. This is not only a multi-tenancy solution but an isolation solution. Isolation is the worse solution for getting multi-tenancy.
          Hide
          toffer Francis Liu added a comment -

          It's adding things that require user configuration. Adding things that require user interaction on failure (failed open is an awful thing to do to a table).

          This is a key feature, it's better to penalize a misbehaving table than have it affect and potentially take down the entire cluster. This will only happen if all the servers are down analogous to a cluster being down. It's pretty intuitive. We can add a better message if need be.

          This adds another table that's required for the master to interact with for assignment and region movement. Right now master is really struggling because of the feedback loop of rit -> meta -> meta moves -> rit. Now we have double the number of tables.

          It actually doesn't. The table is only read at startup and never again.

          On top of that it adds in more things to Zk while we're trying to remove as much data from there as possible.

          Yes, this will be addressed just like the other modules that currently use zk.

          This patch breaks ipv6.
          This patch has Copy Paste code.
          This patch doesn't have the correct headers.
          This patch doesn't load meta's hri from disk on move.

          Thanks for the review. I'll make sure to address them. Can you point out which code is copy paste. Also please elaborate on meta hri? Prolly post it on RB so we don't pollute this discussion?

          Not really they both are user configured places to put regions. Both features interact with the balancer and AM. Both features affect the cluster in disasters in basically the same way.

          IMHO this is a huge oversimplification. I agree with Andrew Purtell let's evaluate regionserver group on it's own merits. ie User configures regionservers to put tables on not regions. Region servers can be any number while in FN it can be at most (or only?) 3. FN was designed to guarantee performance, RS was meant to guarantee isolation.

          There are tons more configurations than that. There's namespace goups, table groups and groups for regionservers to be part of.

          Right now there's only group and it can have regionservers and tables as members.

          Then there's all of the different configurations for the servers that are in different groups.

          This is no different from configuring different clusters and is optional. A really good feature IMHO.

          This is the worst possible way to do multi-tenancy. We're layering a hack on rather than do the real thing.
          You're getting the same number of SPOF's and an added operational complexity.

          IMHO the operational complexity is better than managing the same number of clusters. I honestly would not like to increase my number of SPOFs that increases the probability of one going down. Also isn't there NN HA already?

          If we want multi-tenancy, then lets do that right. We shouldn't accept hacks that don't help most users and make getting to the correct solution harder.

          IMHO this will help a lot of users. This is not only a multi-tenancy solution but an isolation solution. It will help them isolate different workloads (ie CD/PROD, batch/realtime). It will also help them isolate system tables becoming unavailable because a user access pattern/data surfaced a bug which caused the RS to die. It also allows you to configure each group differently (ie GC settings). It also provides coprocessor isolation.

          Show
          toffer Francis Liu added a comment - It's adding things that require user configuration. Adding things that require user interaction on failure (failed open is an awful thing to do to a table). This is a key feature, it's better to penalize a misbehaving table than have it affect and potentially take down the entire cluster. This will only happen if all the servers are down analogous to a cluster being down. It's pretty intuitive. We can add a better message if need be. This adds another table that's required for the master to interact with for assignment and region movement. Right now master is really struggling because of the feedback loop of rit -> meta -> meta moves -> rit. Now we have double the number of tables. It actually doesn't. The table is only read at startup and never again. On top of that it adds in more things to Zk while we're trying to remove as much data from there as possible. Yes, this will be addressed just like the other modules that currently use zk. This patch breaks ipv6. This patch has Copy Paste code. This patch doesn't have the correct headers. This patch doesn't load meta's hri from disk on move. Thanks for the review. I'll make sure to address them. Can you point out which code is copy paste. Also please elaborate on meta hri? Prolly post it on RB so we don't pollute this discussion? Not really they both are user configured places to put regions. Both features interact with the balancer and AM. Both features affect the cluster in disasters in basically the same way. IMHO this is a huge oversimplification. I agree with Andrew Purtell let's evaluate regionserver group on it's own merits. ie User configures regionservers to put tables on not regions. Region servers can be any number while in FN it can be at most (or only?) 3. FN was designed to guarantee performance, RS was meant to guarantee isolation. There are tons more configurations than that. There's namespace goups, table groups and groups for regionservers to be part of. Right now there's only group and it can have regionservers and tables as members. Then there's all of the different configurations for the servers that are in different groups. This is no different from configuring different clusters and is optional. A really good feature IMHO. This is the worst possible way to do multi-tenancy. We're layering a hack on rather than do the real thing. You're getting the same number of SPOF's and an added operational complexity. IMHO the operational complexity is better than managing the same number of clusters. I honestly would not like to increase my number of SPOFs that increases the probability of one going down. Also isn't there NN HA already? If we want multi-tenancy, then lets do that right. We shouldn't accept hacks that don't help most users and make getting to the correct solution harder. IMHO this will help a lot of users. This is not only a multi-tenancy solution but an isolation solution. It will help them isolate different workloads (ie CD/PROD, batch/realtime). It will also help them isolate system tables becoming unavailable because a user access pattern/data surfaced a bug which caused the RS to die. It also allows you to configure each group differently (ie GC settings). It also provides coprocessor isolation.
          Hide
          eclark Elliott Clark added a comment -

          we have other users that will run with RS groups once it is in.

          We're failing all of those users then.

          Memcache based block cache is only used by FB and it is just added complexity for remaining users, but we have it regardless.

          It's a pluggable thing with it off and no added complexity to the default path. If you think it should be in a different module I would be amenable to moving it.
          As i've said before that's what I think we should do here. Make it pluggable with no changes to the core and I'm fine with it.

          Show
          eclark Elliott Clark added a comment - we have other users that will run with RS groups once it is in. We're failing all of those users then. Memcache based block cache is only used by FB and it is just added complexity for remaining users, but we have it regardless. It's a pluggable thing with it off and no added complexity to the default path. If you think it should be in a different module I would be amenable to moving it. As i've said before that's what I think we should do here. Make it pluggable with no changes to the core and I'm fine with it.
          Hide
          enis Enis Soztutar added a comment -

          This patch breaks ipv6.

          This patch has Copy Paste code.

          This patch doesn't have the correct headers.

          This patch doesn't load meta's hri from disk on move.

          These are review comments that are not related to the discussion right now.

          If we want multi-tenancy, then lets do that right. We shouldn't accept hacks that don't help most users and make getting to the correct solution harder.

          I don't understand your reasoning. Yahoo has been running with this for >1 year, and we have other users that will run with RS groups once it is in. The way I see it is not a hack, but a step towards better guarantees. Multi-tenancy and isolation will never be perfect and will always be an ongoing effort to improve it. For example, without making the coprocessors run in a colocated container, we cannot isolate one tenants co-processors from the others today. Saying that we should fix EVERY multitenancy/isolation problem which can take years is not realistic.

          Agreed the ops impact should be totally optional.

          +1 to this. I think with the patch it is already the case. If you are not using the feature, you should not be affected in any way. Yes, with RS groups, if you do not know what you are doing, it is more states and more things can go wrong, but we are not making it default or anything. Memcache based block cache is only used by FB and it is just added complexity for remaining users, but we have it regardless.

          Show
          enis Enis Soztutar added a comment - This patch breaks ipv6. This patch has Copy Paste code. This patch doesn't have the correct headers. This patch doesn't load meta's hri from disk on move. These are review comments that are not related to the discussion right now. If we want multi-tenancy, then lets do that right. We shouldn't accept hacks that don't help most users and make getting to the correct solution harder. I don't understand your reasoning. Yahoo has been running with this for >1 year, and we have other users that will run with RS groups once it is in. The way I see it is not a hack, but a step towards better guarantees. Multi-tenancy and isolation will never be perfect and will always be an ongoing effort to improve it. For example, without making the coprocessors run in a colocated container, we cannot isolate one tenants co-processors from the others today. Saying that we should fix EVERY multitenancy/isolation problem which can take years is not realistic. Agreed the ops impact should be totally optional. +1 to this. I think with the patch it is already the case. If you are not using the feature, you should not be affected in any way. Yes, with RS groups, if you do not know what you are doing, it is more states and more things can go wrong, but we are not making it default or anything. Memcache based block cache is only used by FB and it is just added complexity for remaining users, but we have it regardless.
          Hide
          eclark Elliott Clark added a comment -

          I also find this a lot less impactful than MOB

          This is touching the code that empirically has been the worst part of hbase, and making it more complex. It's adding things that require user configuration. Adding things that require user interaction on failure (failed open is an awful thing to do to a table).

          This adds another table that's required for the master to interact with for assignment and region movement. Right now master is really struggling because of the feedback loop of rit -> meta -> meta moves -> rit. Now we have double the number of tables.

          On top of that it adds in more things to Zk while we're trying to remove as much data from there as possible.

          This patch breaks ipv6.
          This patch has Copy Paste code.
          This patch doesn't have the correct headers.
          This patch doesn't load meta's hri from disk on move.

          IMHO that's not an objective comparison. Favored Nodes and Region Server groups are very different.

          Not really they both are user configured places to put regions. Both features interact with the balancer and AM. Both features affect the cluster in disasters in basically the same way.

          Adding more states?

          Failed open due to bogus servers. So an old state used in a new way. That means something different than it did before.

          That there's an option to configure a different balancer?

          There are tons more configurations than that. There's namespace goups, table groups and groups for regionservers to be part of. Then there's all of the different configurations for the servers that are in different groups.

          This is the worst possible way to do multi-tenancy. We're layering a hack on rather than do the real thing.
          You're getting the same number of SPOF's and an added operational complexity.

          If we want multi-tenancy, then lets do that right. We shouldn't accept hacks that don't help most users and make getting to the correct solution harder.

          Show
          eclark Elliott Clark added a comment - I also find this a lot less impactful than MOB This is touching the code that empirically has been the worst part of hbase, and making it more complex. It's adding things that require user configuration. Adding things that require user interaction on failure (failed open is an awful thing to do to a table). This adds another table that's required for the master to interact with for assignment and region movement. Right now master is really struggling because of the feedback loop of rit -> meta -> meta moves -> rit. Now we have double the number of tables. On top of that it adds in more things to Zk while we're trying to remove as much data from there as possible. This patch breaks ipv6. This patch has Copy Paste code. This patch doesn't have the correct headers. This patch doesn't load meta's hri from disk on move. IMHO that's not an objective comparison. Favored Nodes and Region Server groups are very different. Not really they both are user configured places to put regions. Both features interact with the balancer and AM. Both features affect the cluster in disasters in basically the same way. Adding more states? Failed open due to bogus servers. So an old state used in a new way. That means something different than it did before. That there's an option to configure a different balancer? There are tons more configurations than that. There's namespace goups, table groups and groups for regionservers to be part of. Then there's all of the different configurations for the servers that are in different groups. This is the worst possible way to do multi-tenancy. We're layering a hack on rather than do the real thing. You're getting the same number of SPOF's and an added operational complexity. If we want multi-tenancy, then lets do that right. We shouldn't accept hacks that don't help most users and make getting to the correct solution harder.
          Hide
          apurtell Andrew Purtell added a comment -

          I also find this a lot less impactful than MOB yet that went in. I think we need to be rational about the risks this change actually imposes.

          Show
          apurtell Andrew Purtell added a comment - I also find this a lot less impactful than MOB yet that went in. I think we need to be rational about the risks this change actually imposes.
          Hide
          apurtell Andrew Purtell added a comment -

          Agreed we should evaluate these changes on their own merit.

          No core changes at all isn't a reasonable precondition because even with implementing as a coprocessor based feature we'd need new hooks. Those hooks would touch the same places. IMHO implementing as a coprocessor based feature makes little sense given the relatively small changes needed. it would unnecessarily pollute our space of admin APIs and shell commands with seperate APIs for optional feature like we have with security.

          Agreed the ops impact should be totally optional. Where these changes modify existing function it's reasonable to make sure the modified behavior all sits behind configuration based conditionals.

          Show
          apurtell Andrew Purtell added a comment - Agreed we should evaluate these changes on their own merit. No core changes at all isn't a reasonable precondition because even with implementing as a coprocessor based feature we'd need new hooks. Those hooks would touch the same places. IMHO implementing as a coprocessor based feature makes little sense given the relatively small changes needed. it would unnecessarily pollute our space of admin APIs and shell commands with seperate APIs for optional feature like we have with security. Agreed the ops impact should be totally optional. Where these changes modify existing function it's reasonable to make sure the modified behavior all sits behind configuration based conditionals.
          Hide
          toffer Francis Liu added a comment -

          All of this has already been tried at FB and it was a mistake. This ends up looking functionally very similar to 0.89-fb's favored nodes. ( Only assign regions to specific set of machines that's configured by the admin ). It's so bad that almost every time we try and solve an issue on a cluster with favored nodes, the first thing we do is turn off the balancer so that we don't have to worry about which nodes are configured to have with regions. That's literally step one of debugging. Turn off this feature.
          We'll have a party when FB no longer has this operational nightmare. I won't sign anyone up for the same. I won't sign myself up for the same.

          IMHO that's not an objective comparison. Favored Nodes and Region Server groups are very different. Their use cases are very different and their implementations are also very different.

          As for how useful it is for us (and potenitally for others), if we actually removed region server groups I'm pretty sure our HBase team and SEs would revolt . If we didn't have this feature we would be managing around 80 hbase clusters right now instead of the 6 multi-tenant cluster we are currently running. Step one of debugging is not turning of the balancer that would make things worse. In fact one of the useful features of region server groups is quickly isolating tables to a new group if they are misbehaving or their workload has changed. This can be done in a few minutes if not seconds.

          Assignment manager is already too complex adding more complexity is awful

          If you look at the patch, the change in AM is an extra 20 lines of code. 6 lines are just bugfixes that should be done to AM anyway and the other 14 lines which is fairly straightforward we can even live without if that's what it takes.

          region movement is already too stateful. Adding more is awful

          Adding more states?

          Configuration of HBase is already way too complex. Multiplying that with multiple groups is awful.

          Not sure what the concern here is? That there's an option to configure a different balancer?

          Show
          toffer Francis Liu added a comment - All of this has already been tried at FB and it was a mistake. This ends up looking functionally very similar to 0.89-fb's favored nodes. ( Only assign regions to specific set of machines that's configured by the admin ). It's so bad that almost every time we try and solve an issue on a cluster with favored nodes, the first thing we do is turn off the balancer so that we don't have to worry about which nodes are configured to have with regions. That's literally step one of debugging. Turn off this feature. We'll have a party when FB no longer has this operational nightmare. I won't sign anyone up for the same. I won't sign myself up for the same. IMHO that's not an objective comparison. Favored Nodes and Region Server groups are very different. Their use cases are very different and their implementations are also very different. As for how useful it is for us (and potenitally for others), if we actually removed region server groups I'm pretty sure our HBase team and SEs would revolt . If we didn't have this feature we would be managing around 80 hbase clusters right now instead of the 6 multi-tenant cluster we are currently running. Step one of debugging is not turning of the balancer that would make things worse. In fact one of the useful features of region server groups is quickly isolating tables to a new group if they are misbehaving or their workload has changed. This can be done in a few minutes if not seconds. Assignment manager is already too complex adding more complexity is awful If you look at the patch, the change in AM is an extra 20 lines of code. 6 lines are just bugfixes that should be done to AM anyway and the other 14 lines which is fairly straightforward we can even live without if that's what it takes. region movement is already too stateful. Adding more is awful Adding more states? Configuration of HBase is already way too complex. Multiplying that with multiple groups is awful. Not sure what the concern here is? That there's an option to configure a different balancer?
          Hide
          eclark Elliott Clark added a comment -

          I'm still officially -1 as long as this is built into the core. 99.99%(assuming 10k hbase users) of HBase users should never ever run something like this. It will make an already very operationally complex system un-workable. Because of that anything that's in the default code, adds to the default admin, and is built in is something I can't see ever being ok with.

          All of this has already been tried at FB and it was a mistake. This ends up looking functionally very similar to 0.89-fb's favored nodes. ( Only assign regions to specific set of machines that's configured by the admin ). It's so bad that almost every time we try and solve an issue on a cluster with favored nodes, the first thing we do is turn off the balancer so that we don't have to worry about which nodes are configured to have with regions. That's literally step one of debugging. Turn off this feature.
          We'll have a party when FB no longer has this operational nightmare. I won't sign anyone up for the same. I won't sign myself up for the same.

          So I'm -1 on anything that I can't completely remove. RM -RF.

          • Assignment manager is already too complex adding more complexity is awful
          • region movement is already too stateful. Adding more is awful
          • configuration of HBase is already way too complex. Multiplying that with multiple groups is awful.
          • Admin already has way too many things for users to do that cause issue. Adding more ways for a cluster to be borked is awful.
          Show
          eclark Elliott Clark added a comment - I'm still officially -1 as long as this is built into the core. 99.99%(assuming 10k hbase users) of HBase users should never ever run something like this. It will make an already very operationally complex system un-workable. Because of that anything that's in the default code, adds to the default admin, and is built in is something I can't see ever being ok with. All of this has already been tried at FB and it was a mistake. This ends up looking functionally very similar to 0.89-fb's favored nodes. ( Only assign regions to specific set of machines that's configured by the admin ). It's so bad that almost every time we try and solve an issue on a cluster with favored nodes, the first thing we do is turn off the balancer so that we don't have to worry about which nodes are configured to have with regions. That's literally step one of debugging. Turn off this feature. We'll have a party when FB no longer has this operational nightmare. I won't sign anyone up for the same. I won't sign myself up for the same. So I'm -1 on anything that I can't completely remove. RM -RF. Assignment manager is already too complex adding more complexity is awful region movement is already too stateful. Adding more is awful configuration of HBase is already way too complex. Multiplying that with multiple groups is awful. Admin already has way too many things for users to do that cause issue. Adding more ways for a cluster to be borked is awful.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12753020/HBASE-6721_12.patch
          against master branch at commit cc1542828de93b8d54cc14497fd5937989ea1b6d.
          ATTACHMENT ID: 12753020

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 33 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/15317//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12753020/HBASE-6721_12.patch against master branch at commit cc1542828de93b8d54cc14497fd5937989ea1b6d. ATTACHMENT ID: 12753020 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 33 new or modified tests. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/15317//console This message is automatically generated.
          Hide
          toffer Francis Liu added a comment -

          Sorry it's not a new RB request I updated the old one. Was thinking wether I should just create a new RB request.

          Show
          toffer Francis Liu added a comment - Sorry it's not a new RB request I updated the old one. Was thinking wether I should just create a new RB request.
          Hide
          toffer Francis Liu added a comment -

          Here's a new RB request:

          https://reviews.apache.org/r/27673/

          Show
          toffer Francis Liu added a comment - Here's a new RB request: https://reviews.apache.org/r/27673/
          Hide
          toffer Francis Liu added a comment -

          Sounds good. I don't have any other changes pending so I'm going to update RB with the new patch on trunk.

          Andrew Purtell thanks for taking care of the backport to 0.98 for the patches.

          Show
          toffer Francis Liu added a comment - Sounds good. I don't have any other changes pending so I'm going to update RB with the new patch on trunk. Andrew Purtell thanks for taking care of the backport to 0.98 for the patches.
          Hide
          apurtell Andrew Purtell added a comment -

          I was thinking review patches on subtasks as normal. Once the work in total is ready to attempt a branch merge we can put up a single merge patch for here and RB. At any time it's possible to generate omnibus patches for RB if that would be helpful.

          Show
          apurtell Andrew Purtell added a comment - I was thinking review patches on subtasks as normal. Once the work in total is ready to attempt a branch merge we can put up a single merge patch for here and RB. At any time it's possible to generate omnibus patches for RB if that would be helpful.
          Hide
          enis Enis Soztutar added a comment -

          Chatted with Andrew Purtell at the powow. We'll be doing rebases when pulling in changes from master/0.98.

          So how are we gonna do reviews? I think only the master patch reviews are fine, but it would be good if we list the RBs here.

          Show
          enis Enis Soztutar added a comment - Chatted with Andrew Purtell at the powow. We'll be doing rebases when pulling in changes from master/0.98. So how are we gonna do reviews? I think only the master patch reviews are fine, but it would be good if we list the RBs here.
          Hide
          toffer Francis Liu added a comment -

          Chatted with Andrew Purtell at the powow. We'll be doing rebases when pulling in changes from master/0.98.

          Show
          toffer Francis Liu added a comment - Chatted with Andrew Purtell at the powow. We'll be doing rebases when pulling in changes from master/0.98.
          Hide
          apurtell Andrew Purtell added a comment -

          Let me know if you'd like to rebase the succeeding patches once the one(s) prior are committed.

          I'm thinking we start carving out work as subtasks, applying patches from the subtasks to hbase-6721 branch, make the 0.98 versions of them and apply them to hbase-6721-0.98 branch. This will make development on these branches almost exactly like dev on a release branch.

          On rebase vs. merge:
          1. I can rebase, fix up for changes, then force push, for both hbase-6721 and hbase-6721-0.98
          2. I can merge master into hbase-6721 (and 0.98 into hbase-6721-0.98), fix up for changes, commit the fixups in a merge commit, and then push the merged result.

          There are pros and cons to either approach. What would work best for you Francis Liu?

          Show
          apurtell Andrew Purtell added a comment - Let me know if you'd like to rebase the succeeding patches once the one(s) prior are committed. I'm thinking we start carving out work as subtasks, applying patches from the subtasks to hbase-6721 branch, make the 0.98 versions of them and apply them to hbase-6721-0.98 branch. This will make development on these branches almost exactly like dev on a release branch. On rebase vs. merge: 1. I can rebase, fix up for changes, then force push, for both hbase-6721 and hbase-6721-0.98 2. I can merge master into hbase-6721 (and 0.98 into hbase-6721-0.98), fix up for changes, commit the fixups in a merge commit, and then push the merged result. There are pros and cons to either approach. What would work best for you Francis Liu ?
          Hide
          toffer Francis Liu added a comment -

          I'll create 98 whenver a trunk patch is committed.

          Show
          toffer Francis Liu added a comment - I'll create 98 whenver a trunk patch is committed.
          Hide
          toffer Francis Liu added a comment -

          Andrew Purtell created the patches in separate jiras. The patches are built on top of each other. The order is: HBASE-14312, HBASE-14232, HBASE-14226. Let me know if you'd like to rebase the succeeding patches once the one(s) prior are committed.

          Show
          toffer Francis Liu added a comment - Andrew Purtell created the patches in separate jiras. The patches are built on top of each other. The order is: HBASE-14312 , HBASE-14232 , HBASE-14226 . Let me know if you'd like to rebase the succeeding patches once the one(s) prior are committed.
          Hide
          apurtell Andrew Purtell added a comment -

          Let me know if you'd like me to break these up into separate patches.

          Yes, let's do that. Now that we have feature branches with the previous patches posted here committed, all new work are addendums. It's best IMHO if the addendums have their own JIRA identifiers so we can collect patches and any discussion for each on separate tickets instead of all lumped in here.

          1. Forward port some changes from 0.98 feature branch

          This needs committing to the master branch only. I suggest opening a subtask for it.

          2. Address CP backward compatibility support

          This needs committing to both master and 0.98 branches. Work should be done on HBASE-14232. Pretty sure we will need two patches, one for master, one for 0.98

          3. Address removal of GroupAdmin and GroupAdminClient and moving the apis to Admin.

          This needs committing to both master and 0.98 branches. Work should be done on HBASE-14226. Pretty sure we will need two patches, one for master, one for 0.98

          Show
          apurtell Andrew Purtell added a comment - Let me know if you'd like me to break these up into separate patches. Yes, let's do that. Now that we have feature branches with the previous patches posted here committed, all new work are addendums. It's best IMHO if the addendums have their own JIRA identifiers so we can collect patches and any discussion for each on separate tickets instead of all lumped in here. 1. Forward port some changes from 0.98 feature branch This needs committing to the master branch only. I suggest opening a subtask for it. 2. Address CP backward compatibility support This needs committing to both master and 0.98 branches. Work should be done on HBASE-14232 . Pretty sure we will need two patches, one for master, one for 0.98 3. Address removal of GroupAdmin and GroupAdminClient and moving the apis to Admin. This needs committing to both master and 0.98 branches. Work should be done on HBASE-14226 . Pretty sure we will need two patches, one for master, one for 0.98
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12752029/HBASE-6721_hbase-6721_addendum.patch
          against master branch at commit 9334a47d4570f8adfc003f0fb2c5969a88c3bba0.
          ATTACHMENT ID: 12752029

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 21 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/15222//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12752029/HBASE-6721_hbase-6721_addendum.patch against master branch at commit 9334a47d4570f8adfc003f0fb2c5969a88c3bba0. ATTACHMENT ID: 12752029 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 21 new or modified tests. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/15222//console This message is automatically generated.
          Hide
          toffer Francis Liu added a comment -

          Andrew Purtell Attached is the addedum patch which addresses three things: 1. Forward port some changes from 0.98 feature branch, 2. Address CP backward compatibility support, 3. Address removal of GroupAdmin and GroupAdminClient and moving the apis to Admin. Let me know if you'd like me to break these up into separate patches. Didn't do that as I thought it'd be more complicated.

          Show
          toffer Francis Liu added a comment - Andrew Purtell Attached is the addedum patch which addresses three things: 1. Forward port some changes from 0.98 feature branch, 2. Address CP backward compatibility support, 3. Address removal of GroupAdmin and GroupAdminClient and moving the apis to Admin. Let me know if you'd like me to break these up into separate patches. Didn't do that as I thought it'd be more complicated.
          Hide
          enis Enis Soztutar added a comment -

          This was based on review comments from Enis Soztutar tho I'm not sure if the comment still stands post-cp implementation. I am fine either way. Should this be in HBaseAdmin then?

          This comes back to the discussion of whether RS groups are coprocessor based or core feature. If a core feature, then GroupAdmin methods should move to Admin interface. Otherwise, a GroupAdmin interface + a way to construct the GroupAdminClient would be the way to go.

          Show
          enis Enis Soztutar added a comment - This was based on review comments from Enis Soztutar tho I'm not sure if the comment still stands post-cp implementation. I am fine either way. Should this be in HBaseAdmin then? This comes back to the discussion of whether RS groups are coprocessor based or core feature. If a core feature, then GroupAdmin methods should move to Admin interface. Otherwise, a GroupAdmin interface + a way to construct the GroupAdminClient would be the way to go.
          Hide
          apurtell Andrew Purtell added a comment -

          The tool says hbase-6721-0.98 is 100% binary compatible with 0.98.13, with some source level incompatibility of mild concern in the MasterObserver interface.

          This is prolly because of the new methods this patch adds to the interface. This shouldn't be a concern then?

          Yes, it will be a concern for things like Apache Phoenix. (See their IndexMasterObserver, etc.) We can handle this by using compatibility helpers that won't attempt to invoke the new APIs on MasterObservers that do not implement them. We do something like this in WALCoprocessorHost.

          Show
          apurtell Andrew Purtell added a comment - The tool says hbase-6721-0.98 is 100% binary compatible with 0.98.13, with some source level incompatibility of mild concern in the MasterObserver interface. This is prolly because of the new methods this patch adds to the interface. This shouldn't be a concern then? Yes, it will be a concern for things like Apache Phoenix. (See their IndexMasterObserver, etc.) We can handle this by using compatibility helpers that won't attempt to invoke the new APIs on MasterObservers that do not implement them. We do something like this in WALCoprocessorHost.
          Hide
          apurtell Andrew Purtell added a comment -

          Do we need GroupAdmin and GroupAdminClient now? A hold over from an initial coprocessor based implementation?

          This was based on review comments from Enis Soztutar tho I'm not sure if the comment still stands post-cp implementation. I am fine either way. Should this be in HBaseAdmin then?

          I opened HBASE-14226 to discuss merging them.

          Show
          apurtell Andrew Purtell added a comment - Do we need GroupAdmin and GroupAdminClient now? A hold over from an initial coprocessor based implementation? This was based on review comments from Enis Soztutar tho I'm not sure if the comment still stands post-cp implementation. I am fine either way. Should this be in HBaseAdmin then? I opened HBASE-14226 to discuss merging them.
          Hide
          toffer Francis Liu added a comment -

          I ran the binary compatibility checker tool comparing hbase-6721-0.98 with 0.98.13. The tool says hbase-6721-0.98 is 100% binary compatible with 0.98.13, with some source level incompatibility of mild concern in the MasterObserver interface.

          This is prolly because of the new methods this patch adds to the interface. This shouldn't be a concern then?

          Do we need GroupAdmin and GroupAdminClient now? A hold over from an initial coprocessor based implementation?

          This was based on review comments from Enis Soztutar tho I'm not sure if the comment still stands post-cp implementation. I am fine either way. Should this be in HBaseAdmin then?

          https://reviews.apache.org/r/27673/diff/2?file=751817#file751817line52
          https://reviews.apache.org/r/27673/diff/2?file=751816#file751816line35

          Show
          toffer Francis Liu added a comment - I ran the binary compatibility checker tool comparing hbase-6721-0.98 with 0.98.13. The tool says hbase-6721-0.98 is 100% binary compatible with 0.98.13, with some source level incompatibility of mild concern in the MasterObserver interface. This is prolly because of the new methods this patch adds to the interface. This shouldn't be a concern then? Do we need GroupAdmin and GroupAdminClient now? A hold over from an initial coprocessor based implementation? This was based on review comments from Enis Soztutar tho I'm not sure if the comment still stands post-cp implementation. I am fine either way. Should this be in HBaseAdmin then? https://reviews.apache.org/r/27673/diff/2?file=751817#file751817line52 https://reviews.apache.org/r/27673/diff/2?file=751816#file751816line35
          Hide
          toffer Francis Liu added a comment -

          Thanks Nick Dimiduk wasn't aware of that feature.

          Show
          toffer Francis Liu added a comment - Thanks Nick Dimiduk wasn't aware of that feature.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12750586/HBASE-6721_0.98_2.patch
          against 0.98 branch at commit 45aafb25b7911b5917ab47133e3e4268806e4c91.
          ATTACHMENT ID: 12750586

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 33 new or modified tests.

          +1 hadoop versions. The patch compiles with all supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0)

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 protoc. The applied patch does not increase the total number of protoc compiler warnings.

          -1 javadoc. The javadoc tool appears to have generated 28 warning messages.

          -1 checkstyle. The applied patch generated 3913 checkstyle errors (more than the master's current 3876 errors).

          -1 InterfaceAudience. The patch appears to contain InterfaceAudience from hadoop rather than hbase:
          +import org.apache.hadoop.classification.InterfaceAudience;
          +import org.apache.hadoop.classification.InterfaceStability;
          +import org.apache.hadoop.classification.InterfaceAudience;
          +import org.apache.hadoop.classification.InterfaceAudience;.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          -1 release audit. The applied patch generated 1 release audit warnings (more than the master's current 0 warnings).

          -1 lineLengths. The patch introduces the following lines longer than 100:
          + public GetGroupInfoResponse getGroupInfo(RpcController controller, GetGroupInfoRequest request) throws ServiceException {
          + public GetGroupInfoOfTableResponse getGroupInfoOfTable(RpcController controller, GetGroupInfoOfTableRequest request) throws ServiceException {
          + public GetGroupInfoOfServerResponse getGroupInfoOfServer(RpcController controller, GetGroupInfoOfServerRequest request) throws ServiceException {
          + public MoveServersResponse moveServers(RpcController controller, MoveServersRequest request) throws ServiceException {
          + public MoveTablesResponse moveTables(RpcController controller, MoveTablesRequest request) throws ServiceException {
          + public AddGroupResponse addGroup(RpcController controller, AddGroupRequest request) throws ServiceException {
          + public RemoveGroupResponse removeGroup(RpcController controller, RemoveGroupRequest request) throws ServiceException {
          + public BalanceGroupResponse balanceGroup(RpcController controller, BalanceGroupRequest request) throws ServiceException {
          + public ListGroupInfosResponse listGroupInfos(RpcController controller, ListGroupInfosRequest request) throws ServiceException {
          + * <code>rpc GetGroupInfoOfTable(.GetGroupInfoOfTableRequest) returns (.GetGroupInfoOfTableResponse);</code>

          -1 site. The patch appears to cause mvn post-site goal to fail.

          +1 core tests. The patch passed unit tests in .

          -1 core zombie tests. There are 5 zombie test(s): at org.apache.hadoop.hbase.regionserver.wal.TestLogRolling.testLogRolling(TestLogRolling.java:219)
          at org.apache.hadoop.hbase.TestIOFencing.testFencingAroundCompaction(TestIOFencing.java:227)
          at org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster.testRITStateForRollback(TestSplitTransactionOnCluster.java:292)
          at org.apache.hadoop.hbase.io.encoding.TestEncodedSeekers.testEncodedSeeker(TestEncodedSeekers.java:118)
          at org.apache.hadoop.hbase.TestClusterBootOrder.testBootRegionServerFirst(TestClusterBootOrder.java:104)

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/15105//testReport/
          Release audit warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/15105//artifact/patchprocess/patchReleaseAuditWarnings.txt
          Release Findbugs (version 2.0.3) warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/15105//artifact/patchprocess/newFindbugsWarnings.html
          Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/15105//artifact/patchprocess/checkstyle-aggregate.html

          Javadoc warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/15105//artifact/patchprocess/patchJavadocWarnings.txt
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/15105//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12750586/HBASE-6721_0.98_2.patch against 0.98 branch at commit 45aafb25b7911b5917ab47133e3e4268806e4c91. ATTACHMENT ID: 12750586 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 33 new or modified tests. +1 hadoop versions . The patch compiles with all supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0) +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 protoc . The applied patch does not increase the total number of protoc compiler warnings. -1 javadoc . The javadoc tool appears to have generated 28 warning messages. -1 checkstyle . The applied patch generated 3913 checkstyle errors (more than the master's current 3876 errors). -1 InterfaceAudience . The patch appears to contain InterfaceAudience from hadoop rather than hbase: +import org.apache.hadoop.classification.InterfaceAudience; +import org.apache.hadoop.classification.InterfaceStability; +import org.apache.hadoop.classification.InterfaceAudience; +import org.apache.hadoop.classification.InterfaceAudience;. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. -1 release audit . The applied patch generated 1 release audit warnings (more than the master's current 0 warnings). -1 lineLengths . The patch introduces the following lines longer than 100: + public GetGroupInfoResponse getGroupInfo(RpcController controller, GetGroupInfoRequest request) throws ServiceException { + public GetGroupInfoOfTableResponse getGroupInfoOfTable(RpcController controller, GetGroupInfoOfTableRequest request) throws ServiceException { + public GetGroupInfoOfServerResponse getGroupInfoOfServer(RpcController controller, GetGroupInfoOfServerRequest request) throws ServiceException { + public MoveServersResponse moveServers(RpcController controller, MoveServersRequest request) throws ServiceException { + public MoveTablesResponse moveTables(RpcController controller, MoveTablesRequest request) throws ServiceException { + public AddGroupResponse addGroup(RpcController controller, AddGroupRequest request) throws ServiceException { + public RemoveGroupResponse removeGroup(RpcController controller, RemoveGroupRequest request) throws ServiceException { + public BalanceGroupResponse balanceGroup(RpcController controller, BalanceGroupRequest request) throws ServiceException { + public ListGroupInfosResponse listGroupInfos(RpcController controller, ListGroupInfosRequest request) throws ServiceException { + * <code>rpc GetGroupInfoOfTable(.GetGroupInfoOfTableRequest) returns (.GetGroupInfoOfTableResponse);</code> -1 site . The patch appears to cause mvn post-site goal to fail. +1 core tests . The patch passed unit tests in . -1 core zombie tests . There are 5 zombie test(s): at org.apache.hadoop.hbase.regionserver.wal.TestLogRolling.testLogRolling(TestLogRolling.java:219) at org.apache.hadoop.hbase.TestIOFencing.testFencingAroundCompaction(TestIOFencing.java:227) at org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster.testRITStateForRollback(TestSplitTransactionOnCluster.java:292) at org.apache.hadoop.hbase.io.encoding.TestEncodedSeekers.testEncodedSeeker(TestEncodedSeekers.java:118) at org.apache.hadoop.hbase.TestClusterBootOrder.testBootRegionServerFirst(TestClusterBootOrder.java:104) Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/15105//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/15105//artifact/patchprocess/patchReleaseAuditWarnings.txt Release Findbugs (version 2.0.3) warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/15105//artifact/patchprocess/newFindbugsWarnings.html Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/15105//artifact/patchprocess/checkstyle-aggregate.html Javadoc warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/15105//artifact/patchprocess/patchJavadocWarnings.txt Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/15105//console This message is automatically generated.
          Hide
          apurtell Andrew Purtell added a comment -

          Do we need GroupAdmin and GroupAdminClient now? A hold over from an initial coprocessor based implementation?

          Show
          apurtell Andrew Purtell added a comment - Do we need GroupAdmin and GroupAdminClient now? A hold over from an initial coprocessor based implementation?
          Hide
          apurtell Andrew Purtell added a comment -

          I ran the binary compatibility checker tool comparing hbase-6721-0.98 with 0.98.13. The tool says hbase-6721-0.98 is 100% binary compatible with 0.98.13, with some source level incompatibility of mild concern in the MasterObserver interface.

          Show
          apurtell Andrew Purtell added a comment - I ran the binary compatibility checker tool comparing hbase-6721-0.98 with 0.98.13. The tool says hbase-6721-0.98 is 100% binary compatible with 0.98.13, with some source level incompatibility of mild concern in the MasterObserver interface.
          Hide
          ndimiduk Nick Dimiduk added a comment -

          Reattaching 'HBASE-6721_0.98_2.patch' with '0.98' in the name to match the branch-specific filter.

          Show
          ndimiduk Nick Dimiduk added a comment - Reattaching ' HBASE-6721 _0.98_2.patch' with '0.98' in the name to match the branch-specific filter.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12750554/HBASE-6721_98_2.patch
          against master branch at commit 45aafb25b7911b5917ab47133e3e4268806e4c91.
          ATTACHMENT ID: 12750554

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 33 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/15104//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12750554/HBASE-6721_98_2.patch against master branch at commit 45aafb25b7911b5917ab47133e3e4268806e4c91. ATTACHMENT ID: 12750554 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 33 new or modified tests. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/15104//console This message is automatically generated.
          Hide
          apurtell Andrew Purtell added a comment -

          the patch should be good to commit.

          I made a branch 'hbase-6721-0.98' with the v2 0.98 patch applied and pushed it.

          will need to create an addendum for trunk patch.

          Please post the addendum here, I'll commit it to hbase-6721 branch.

          When updating the feature branches do you prefer me to merge or rebase? I typically rebase in my own work but will do what you prefer, just say.

          Show
          apurtell Andrew Purtell added a comment - the patch should be good to commit. I made a branch 'hbase-6721-0.98' with the v2 0.98 patch applied and pushed it. will need to create an addendum for trunk patch. Please post the addendum here, I'll commit it to hbase-6721 branch. When updating the feature branches do you prefer me to merge or rebase? I typically rebase in my own work but will do what you prefer, just say.
          Hide
          toffer Francis Liu added a comment -

          Tests passed tho I had to patch HBase/IntegrationTestingUtilitily to get the tests to run properly, there was a regression in an api that used to work in distributed mode and now it doesn't. Will deal with that issue in a separate jira.

          Reconciled some changes with internal implementation, rebased.

          Andrew Purtell the patch should be good to commit. will need to create an addendum for trunk patch.

          Show
          toffer Francis Liu added a comment - Tests passed tho I had to patch HBase/IntegrationTestingUtilitily to get the tests to run properly, there was a regression in an api that used to work in distributed mode and now it doesn't. Will deal with that issue in a separate jira. Reconciled some changes with internal implementation, rebased. Andrew Purtell the patch should be good to commit. will need to create an addendum for trunk patch.
          Hide
          apurtell Andrew Purtell added a comment -

          draft of backported patch to 98. need to run integration and unit tests then it should be good to go.

          Thanks Francis Liu, let me know when you think it's ready to go in and I'll make a branch with its application

          Show
          apurtell Andrew Purtell added a comment - draft of backported patch to 98. need to run integration and unit tests then it should be good to go. Thanks Francis Liu , let me know when you think it's ready to go in and I'll make a branch with its application
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12750267/HBASE-6721_98_1.patch
          against master branch at commit 9c69bf766fcad024bef5760f242cae2bc609b374.
          ATTACHMENT ID: 12750267

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 33 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/15084//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12750267/HBASE-6721_98_1.patch against master branch at commit 9c69bf766fcad024bef5760f242cae2bc609b374. ATTACHMENT ID: 12750267 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 33 new or modified tests. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/15084//console This message is automatically generated.
          Hide
          toffer Francis Liu added a comment -

          1. Within the interface GroupInfoManager.java, I noticed that the function getGroupOfServer returns a GroupInfo object, but the function getGroupOfTable returns a String object. Was there a performance consideration or some other reason for returning a string? (It seems the API would be more consistent to return a GroupInfo object.)

          Yeah performance. I didn't want to have to gather the information to generate the GroupInfo object since most internal calls won't need it. The external api call is consistent tho.

          2. For the function onlineRetainAssignment why are regions assigned to bogus so it ends up in RIT if a server is not available? (We would like to keep as few regions in RIT as possible in order to maximize our availability.)

          This guarantees isolation. Which is one of the key features of Region Server Groups. ie you don't want the reason that the region from one group to run out of live servers to affect the regions in another group. So worst case you affect the availability of the group and not the entire cluster. What's your use case for region server groups?

          3. On the topic of onlineRetainAssignment, what is the objective for separating online and offline servers? I noticed that another balancer such as the StochasticLoadBalancer does not make such a distinction.

          Yes that's inherent in GroupBaseLoadBalancer. Stochastic only looks at what's online, while Group needs to look at which member servers are online.

          Show
          toffer Francis Liu added a comment - 1. Within the interface GroupInfoManager.java, I noticed that the function getGroupOfServer returns a GroupInfo object, but the function getGroupOfTable returns a String object. Was there a performance consideration or some other reason for returning a string? (It seems the API would be more consistent to return a GroupInfo object.) Yeah performance. I didn't want to have to gather the information to generate the GroupInfo object since most internal calls won't need it. The external api call is consistent tho. 2. For the function onlineRetainAssignment why are regions assigned to bogus so it ends up in RIT if a server is not available? (We would like to keep as few regions in RIT as possible in order to maximize our availability.) This guarantees isolation. Which is one of the key features of Region Server Groups. ie you don't want the reason that the region from one group to run out of live servers to affect the regions in another group. So worst case you affect the availability of the group and not the entire cluster. What's your use case for region server groups? 3. On the topic of onlineRetainAssignment, what is the objective for separating online and offline servers? I noticed that another balancer such as the StochasticLoadBalancer does not make such a distinction. Yes that's inherent in GroupBaseLoadBalancer. Stochastic only looks at what's online, while Group needs to look at which member servers are online.
          Hide
          toffer Francis Liu added a comment -

          draft of backported patch to 98. need to run integration and unit tests then it should be good to go.

          Show
          toffer Francis Liu added a comment - draft of backported patch to 98. need to run integration and unit tests then it should be good to go.
          Hide
          gwu91 Guorui Wu added a comment -

          As I've been studying this code, in order to aid with the understanding of HBASE-6721, I’ve created some UML sequence diagrams of functions within GroupBasedLoadBalancer.java. This UML diagram only focuses on GroupBasedLoadBalancer, since GroupBasedLoadBalancer, along with GroupInfoManager, GroupInfoManagerImpl and GroupInfo as well as some way to configure groups represent the core functionality needed for this implementation. This represents the 1,500 core lines implementing the region server group functionality; this documentation does not cover the extensive configuration management via CLI stored in Zookeeper and an HBase table.

          The goal of GroupBasedLoadBalancer is to separate servers and regions into groups, and to balance within groups using another load balancer (referred to as internal load balancer). The default load balancer used is StochasticLoadBalancer. Each group is stored as a GroupInfo object, and GroupInfoManagerImpl holds a collection of GroupInfo objects with APIs allowing for the manipulation of this collection. These groups can be created and configured via the CLI, and are stored within an HBase table as well as Zookeeper. As GroupBasedLoadBalancer implements the LoadBalancer interface, it has the following functions from the interface:
          As GroupBasedLoadBalancer implements the LoadBalancer interface, it has the following functions from the interface:
          • balanceCluster
          • roundRobinAssignment
          • retainAssignment
          • immediateAssignment
          • randomAssignment
          • initialize
          • isStopped
          • setGroupInfoManager

          Additionally, GroupBasedLoadBalancer has several helper functions:
          • offlineRetainAssignment
          • onlineRetainAssignment
          • generateGroupMaps
          • filterOfflineServers
          • filterServers
          • getMisplacedRegions
          • correctAssignments
          • isOnline
          • getGroupInfoManager

          Through reading the code for the load balancer portion of HBASE-6721, I have a few questions:
          1. Within the interface GroupInfoManager.java, I noticed that the function getGroupOfServer returns a GroupInfo object, but the function getGroupOfTable returns a String object. Was there a performance consideration or some other reason for returning a string? (It seems the API would be more consistent to return a GroupInfo object.)
          2. For the function onlineRetainAssignment why are regions assigned to bogus so it ends up in RIT if a server is not available? (We would like to keep as few regions in RIT as possible in order to maximize our availability.)
          3. On the topic of onlineRetainAssignment, what is the objective for separating online and offline servers? I noticed that another balancer such as the StochasticLoadBalancer does not make such a distinction.

          Lastly, the UML diagram I created can be edited by downloading the attached XML file and editing with http://draw.io.

          Show
          gwu91 Guorui Wu added a comment - As I've been studying this code, in order to aid with the understanding of HBASE-6721 , I’ve created some UML sequence diagrams of functions within GroupBasedLoadBalancer.java. This UML diagram only focuses on GroupBasedLoadBalancer, since GroupBasedLoadBalancer, along with GroupInfoManager, GroupInfoManagerImpl and GroupInfo as well as some way to configure groups represent the core functionality needed for this implementation. This represents the 1,500 core lines implementing the region server group functionality; this documentation does not cover the extensive configuration management via CLI stored in Zookeeper and an HBase table. The goal of GroupBasedLoadBalancer is to separate servers and regions into groups, and to balance within groups using another load balancer (referred to as internal load balancer). The default load balancer used is StochasticLoadBalancer . Each group is stored as a GroupInfo object, and GroupInfoManagerImpl holds a collection of GroupInfo objects with APIs allowing for the manipulation of this collection. These groups can be created and configured via the CLI, and are stored within an HBase table as well as Zookeeper. As GroupBasedLoadBalancer implements the LoadBalancer interface, it has the following functions from the interface: As GroupBasedLoadBalancer implements the LoadBalancer interface, it has the following functions from the interface: • balanceCluster • roundRobinAssignment • retainAssignment • immediateAssignment • randomAssignment • initialize • isStopped • setGroupInfoManager Additionally, GroupBasedLoadBalancer has several helper functions: • offlineRetainAssignment • onlineRetainAssignment • generateGroupMaps • filterOfflineServers • filterServers • getMisplacedRegions • correctAssignments • isOnline • getGroupInfoManager Through reading the code for the load balancer portion of HBASE-6721 , I have a few questions: 1. Within the interface GroupInfoManager.java, I noticed that the function getGroupOfServer returns a GroupInfo object, but the function getGroupOfTable returns a String object. Was there a performance consideration or some other reason for returning a string? (It seems the API would be more consistent to return a GroupInfo object.) 2. For the function onlineRetainAssignment why are regions assigned to bogus so it ends up in RIT if a server is not available? (We would like to keep as few regions in RIT as possible in order to maximize our availability.) 3. On the topic of onlineRetainAssignment, what is the objective for separating online and offline servers? I noticed that another balancer such as the StochasticLoadBalancer does not make such a distinction. Lastly, the UML diagram I created can be edited by downloading the attached XML file and editing with http://draw.io .
          Hide
          toffer Francis Liu added a comment -

          Thanks Andrew Purtell, that'd be great let me work on one.

          Show
          toffer Francis Liu added a comment - Thanks Andrew Purtell , that'd be great let me work on one.
          Hide
          apurtell Andrew Purtell added a comment - - edited

          Thanks Francis Liu. I applied your latest patch to master and pushed the result as new branch "hbase-6721". I checked that it compiled before pushing but didn't run tests.

          Do you need this for 0.98? If so, we can do another branch "hbase-6721-0.98" with a backport and rebase it at every RC. Or branch-1.

          Show
          apurtell Andrew Purtell added a comment - - edited Thanks Francis Liu . I applied your latest patch to master and pushed the result as new branch "hbase-6721". I checked that it compiled before pushing but didn't run tests. Do you need this for 0.98? If so, we can do another branch "hbase-6721-0.98" with a backport and rebase it at every RC. Or branch-1.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12747704/HBASE-6721_11.patch
          against master branch at commit 05de2ec5801fbba4577fb363f858a6e6f282c104.
          ATTACHMENT ID: 12747704

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 30 new or modified tests.

          +1 hadoop versions. The patch compiles with all supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0)

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 protoc. The applied patch does not increase the total number of protoc compiler warnings.

          -1 javadoc. The javadoc tool appears to have generated 6 warning messages.

          -1 checkstyle. The applied patch generated 1908 checkstyle errors (more than the master's current 1864 errors).

          -1 InterfaceAudience. The patch appears to contain InterfaceAudience from hadoop rather than hbase:
          +import org.apache.hadoop.classification.InterfaceAudience;
          +import org.apache.hadoop.classification.InterfaceStability;
          +import org.apache.hadoop.classification.InterfaceAudience;
          +import org.apache.hadoop.classification.InterfaceAudience;.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          -1 release audit. The applied patch generated 1 release audit warnings (more than the master's current 0 warnings).

          -1 lineLengths. The patch introduces the following lines longer than 100:
          + * <code>rpc getLastMajorCompactionTimestamp(.hbase.pb.MajorCompactionTimestampRequest) returns (.hbase.pb.MajorCompactionTimestampResponse);</code>
          + * <code>rpc getLastMajorCompactionTimestampForRegion(.hbase.pb.MajorCompactionTimestampForRegionRequest) returns (.hbase.pb.MajorCompactionTimestampResponse);</code>
          + * <code>rpc getProcedureResult(.hbase.pb.GetProcedureResultRequest) returns (.hbase.pb.GetProcedureResultResponse);</code>
          + * <code>rpc GetGroupInfo(.hbase.pb.GetGroupInfoRequest) returns (.hbase.pb.GetGroupInfoResponse);</code>
          + * <code>rpc GetGroupInfoOfTable(.hbase.pb.GetGroupInfoOfTableRequest) returns (.hbase.pb.GetGroupInfoOfTableResponse);</code>
          + * <code>rpc GetGroupInfoOfServer(.hbase.pb.GetGroupInfoOfServerRequest) returns (.hbase.pb.GetGroupInfoOfServerResponse);</code>
          + * <code>rpc MoveServers(.hbase.pb.MoveServersRequest) returns (.hbase.pb.MoveServersResponse);</code>
          + * <code>rpc MoveTables(.hbase.pb.MoveTablesRequest) returns (.hbase.pb.MoveTablesResponse);</code>
          + * <code>rpc RemoveGroup(.hbase.pb.RemoveGroupRequest) returns (.hbase.pb.RemoveGroupResponse);</code>
          + * <code>rpc BalanceGroup(.hbase.pb.BalanceGroupRequest) returns (.hbase.pb.BalanceGroupResponse);</code>

          +1 site. The mvn post-site goal succeeds with this patch.

          -1 core tests. The patch failed these unit tests:

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/14922//testReport/
          Release audit warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/14922//artifact/patchprocess/patchReleaseAuditWarnings.txt
          Release Findbugs (version 2.0.3) warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/14922//artifact/patchprocess/newFindbugsWarnings.html
          Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/14922//artifact/patchprocess/checkstyle-aggregate.html

          Javadoc warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/14922//artifact/patchprocess/patchJavadocWarnings.txt
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/14922//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12747704/HBASE-6721_11.patch against master branch at commit 05de2ec5801fbba4577fb363f858a6e6f282c104. ATTACHMENT ID: 12747704 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 30 new or modified tests. +1 hadoop versions . The patch compiles with all supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0) +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 protoc . The applied patch does not increase the total number of protoc compiler warnings. -1 javadoc . The javadoc tool appears to have generated 6 warning messages. -1 checkstyle . The applied patch generated 1908 checkstyle errors (more than the master's current 1864 errors). -1 InterfaceAudience . The patch appears to contain InterfaceAudience from hadoop rather than hbase: +import org.apache.hadoop.classification.InterfaceAudience; +import org.apache.hadoop.classification.InterfaceStability; +import org.apache.hadoop.classification.InterfaceAudience; +import org.apache.hadoop.classification.InterfaceAudience;. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. -1 release audit . The applied patch generated 1 release audit warnings (more than the master's current 0 warnings). -1 lineLengths . The patch introduces the following lines longer than 100: + * <code>rpc getLastMajorCompactionTimestamp(.hbase.pb.MajorCompactionTimestampRequest) returns (.hbase.pb.MajorCompactionTimestampResponse);</code> + * <code>rpc getLastMajorCompactionTimestampForRegion(.hbase.pb.MajorCompactionTimestampForRegionRequest) returns (.hbase.pb.MajorCompactionTimestampResponse);</code> + * <code>rpc getProcedureResult(.hbase.pb.GetProcedureResultRequest) returns (.hbase.pb.GetProcedureResultResponse);</code> + * <code>rpc GetGroupInfo(.hbase.pb.GetGroupInfoRequest) returns (.hbase.pb.GetGroupInfoResponse);</code> + * <code>rpc GetGroupInfoOfTable(.hbase.pb.GetGroupInfoOfTableRequest) returns (.hbase.pb.GetGroupInfoOfTableResponse);</code> + * <code>rpc GetGroupInfoOfServer(.hbase.pb.GetGroupInfoOfServerRequest) returns (.hbase.pb.GetGroupInfoOfServerResponse);</code> + * <code>rpc MoveServers(.hbase.pb.MoveServersRequest) returns (.hbase.pb.MoveServersResponse);</code> + * <code>rpc MoveTables(.hbase.pb.MoveTablesRequest) returns (.hbase.pb.MoveTablesResponse);</code> + * <code>rpc RemoveGroup(.hbase.pb.RemoveGroupRequest) returns (.hbase.pb.RemoveGroupResponse);</code> + * <code>rpc BalanceGroup(.hbase.pb.BalanceGroupRequest) returns (.hbase.pb.BalanceGroupResponse);</code> +1 site . The mvn post-site goal succeeds with this patch. -1 core tests . The patch failed these unit tests: Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/14922//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/14922//artifact/patchprocess/patchReleaseAuditWarnings.txt Release Findbugs (version 2.0.3) warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/14922//artifact/patchprocess/newFindbugsWarnings.html Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/14922//artifact/patchprocess/checkstyle-aggregate.html Javadoc warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/14922//artifact/patchprocess/patchJavadocWarnings.txt Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/14922//console This message is automatically generated.
          Hide
          toffer Francis Liu added a comment -

          Andrew Purtell I rebased the patch onto trunk. This should be good enough to commit to a branch.

          Show
          toffer Francis Liu added a comment - Andrew Purtell I rebased the patch onto trunk. This should be good enough to commit to a branch.
          Hide
          toffer Francis Liu added a comment -

          Rebased patch.

          Show
          toffer Francis Liu added a comment - Rebased patch.
          Hide
          toffer Francis Liu added a comment -

          Thanks Andrew Purtell let me go through the patch and rebase it as well. Will put up an updated patch this week.

          Show
          toffer Francis Liu added a comment - Thanks Andrew Purtell let me go through the patch and rebase it as well. Will put up an updated patch this week.
          Hide
          apurtell Andrew Purtell added a comment -

          Let me suggest that we commit what we have with a cursory review to a branch and then make progress there.

          +1
          Ping me for assistance when you're ready to commit to a branch Francis Liu

          Show
          apurtell Andrew Purtell added a comment - Let me suggest that we commit what we have with a cursory review to a branch and then make progress there. +1 Ping me for assistance when you're ready to commit to a branch Francis Liu
          Hide
          jmhsieh Jonathan Hsieh added a comment -

          Sorry, I didn't realize I was blocking. I can't commit to doing a review in the short term – please proceed without me. Let me suggest that we commit what we have with a cursory review to a branch and then make progress there.

          Show
          jmhsieh Jonathan Hsieh added a comment - Sorry, I didn't realize I was blocking. I can't commit to doing a review in the short term – please proceed without me. Let me suggest that we commit what we have with a cursory review to a branch and then make progress there.
          Hide
          toffer Francis Liu added a comment -

          Unless I'm missing something it seems the review is still pending.

          Jonathan Hsieh would you still be able to complete the review?

          Show
          toffer Francis Liu added a comment - Unless I'm missing something it seems the review is still pending. Jonathan Hsieh would you still be able to complete the review?
          Hide
          apurtell Andrew Purtell added a comment -

          Creating the branch is easy. Was the review on RB finished?

          Show
          apurtell Andrew Purtell added a comment - Creating the branch is easy. Was the review on RB finished?
          Hide
          yuzhihong@gmail.com Ted Yu added a comment -

          Looks like branch hbase-6721 has not been created.

          Should we proceed with what Jon suggested above ?

          Show
          yuzhihong@gmail.com Ted Yu added a comment - Looks like branch hbase-6721 has not been created. Should we proceed with what Jon suggested above ?
          Hide
          toffer Francis Liu added a comment -

          Works for me.

          Show
          toffer Francis Liu added a comment - Works for me.
          Hide
          jmhsieh Jonathan Hsieh added a comment -

          Francis Liu. Here's my suggestion: Let's finish the current review in review board. Then instead of rebasing, let's use that patch from the point you branched at and start a 'hbase-6721' from there. We can do periodic merges into the 'hbase-6721' branch afterwards and capture the changes due to the merges.

          Show
          jmhsieh Jonathan Hsieh added a comment - Francis Liu . Here's my suggestion: Let's finish the current review in review board. Then instead of rebasing, let's use that patch from the point you branched at and start a 'hbase-6721' from there. We can do periodic merges into the 'hbase-6721' branch afterwards and capture the changes due to the merges.
          Hide
          toffer Francis Liu added a comment -

          We discussed this during HBaseCon. The consensus was to get this into a feature branch to address any stability/integration issues as well as any other concerns.

          Jonathan Hsieh Elliott Clark Enis Soztutar How to we go about creating the branch? Should we finish the review and merge it in? Should I rebase?