Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-6775

Use ZK.multi when available for HBASE-6710 0.92/0.94 compatibility fix

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 0.94.2
    • Fix Version/s: 0.94.4
    • Component/s: Zookeeper
    • Labels:
      None
    • Release Note:
      Hide
      hbase.zookeeper.useMulti
      Instructs HBase to make use of ZooKeeper's multi-update functionality.
      + This allows certain ZooKeeper operations to complete more quickly and prevents some issues
      + with rare ZooKeeper failure scenarios (see the release note of HBASE-6710 for an example).
      + IMPORTANT: only set this to true if all ZooKeeper servers in the cluster are on version 3.4+
      + and will not be downgraded. ZooKeeper versions before 3.4 do not support multi-update and will
      + not fail gracefully if multi-update is invoked (see ZOOKEEPER-1495).
      Show
      hbase.zookeeper.useMulti Instructs HBase to make use of ZooKeeper's multi-update functionality. + This allows certain ZooKeeper operations to complete more quickly and prevents some issues + with rare ZooKeeper failure scenarios (see the release note of HBASE-6710 for an example). + IMPORTANT: only set this to true if all ZooKeeper servers in the cluster are on version 3.4+ + and will not be downgraded. ZooKeeper versions before 3.4 do not support multi-update and will + not fail gracefully if multi-update is invoked (see ZOOKEEPER-1495 ).

      Description

      This issue introduces the ability for the HMaster to make use of ZooKeeper's multi-update functionality. This allows certain ZooKeeper operations to complete more quickly and prevents some issues with rare ZooKeeper failure scenarios (see the release note of HBASE-6710 for an example). This feature is off by default; to enable set "hbase.zookeeper.useMulti" to true in the configuration of the HMaster.

      IMPORTANT: "hbase.zookeeper.useMulti" should only be set to true if all ZooKeeper servers in the cluster are on version 3.4+ and will not be downgraded. ZooKeeper versions before 3.4 do not support multi-update and will not fail gracefully if multi-update is invoked (see ZOOKEEPER-1495).

      1. HBASE-6775-v2.patch
        40 kB
        Gregory Chanan

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #374 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/374/)
          HBASE-7382 Port ZK.multi support from HBASE-6775 to 0.96 (Gregory, Himanshu and Ted) (Revision 1438317)

          Result = FAILURE
          tedyu :
          Files :

          • /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
          • /hbase/trunk/hbase-server/src/main/resources/hbase-default.xml
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKMulti.java
          Show
          hudson Hudson added a comment - Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #374 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/374/ ) HBASE-7382 Port ZK.multi support from HBASE-6775 to 0.96 (Gregory, Himanshu and Ted) (Revision 1438317) Result = FAILURE tedyu : Files : /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java /hbase/trunk/hbase-server/src/main/resources/hbase-default.xml /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKMulti.java
          Hide
          hudson Hudson added a comment -

          Integrated in HBase-TRUNK #3793 (See https://builds.apache.org/job/HBase-TRUNK/3793/)
          HBASE-7382 Port ZK.multi support from HBASE-6775 to 0.96 (Gregory, Himanshu and Ted) (Revision 1438317)

          Result = FAILURE
          tedyu :
          Files :

          • /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
          • /hbase/trunk/hbase-server/src/main/resources/hbase-default.xml
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKMulti.java
          Show
          hudson Hudson added a comment - Integrated in HBase-TRUNK #3793 (See https://builds.apache.org/job/HBase-TRUNK/3793/ ) HBASE-7382 Port ZK.multi support from HBASE-6775 to 0.96 (Gregory, Himanshu and Ted) (Revision 1438317) Result = FAILURE tedyu : Files : /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java /hbase/trunk/hbase-server/src/main/resources/hbase-default.xml /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKMulti.java
          Hide
          hudson Hudson added a comment -

          Integrated in HBase-0.94-security-on-Hadoop-23 #10 (See https://builds.apache.org/job/HBase-0.94-security-on-Hadoop-23/10/)
          HBASE-6775 Use ZK.multi when available (Addendum: add test file) (Revision 1423635)
          HBASE-6775 Use ZK.multi when available for HBASE-6710 0.92/0.94 compatibility fix (Revision 1423616)

          Result = FAILURE
          gchanan :
          Files :

          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKMulti.java

          gchanan :
          Files :

          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/HConstants.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKTable.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
          • /hbase/branches/0.94/src/main/resources/hbase-default.xml
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKTable.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKTableReadOnly.java
          Show
          hudson Hudson added a comment - Integrated in HBase-0.94-security-on-Hadoop-23 #10 (See https://builds.apache.org/job/HBase-0.94-security-on-Hadoop-23/10/ ) HBASE-6775 Use ZK.multi when available (Addendum: add test file) (Revision 1423635) HBASE-6775 Use ZK.multi when available for HBASE-6710 0.92/0.94 compatibility fix (Revision 1423616) Result = FAILURE gchanan : Files : /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKMulti.java gchanan : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/HConstants.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKTable.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java /hbase/branches/0.94/src/main/resources/hbase-default.xml /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKTable.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKTableReadOnly.java
          Hide
          hudson Hudson added a comment -

          Integrated in HBase-0.94-security #87 (See https://builds.apache.org/job/HBase-0.94-security/87/)
          HBASE-6775 Use ZK.multi when available (Addendum: add test file) (Revision 1423635)
          HBASE-6775 Use ZK.multi when available for HBASE-6710 0.92/0.94 compatibility fix (Revision 1423616)

          Result = SUCCESS
          gchanan :
          Files :

          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKMulti.java

          gchanan :
          Files :

          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/HConstants.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKTable.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
          • /hbase/branches/0.94/src/main/resources/hbase-default.xml
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKTable.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKTableReadOnly.java
          Show
          hudson Hudson added a comment - Integrated in HBase-0.94-security #87 (See https://builds.apache.org/job/HBase-0.94-security/87/ ) HBASE-6775 Use ZK.multi when available (Addendum: add test file) (Revision 1423635) HBASE-6775 Use ZK.multi when available for HBASE-6710 0.92/0.94 compatibility fix (Revision 1423616) Result = SUCCESS gchanan : Files : /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKMulti.java gchanan : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/HConstants.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKTable.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java /hbase/branches/0.94/src/main/resources/hbase-default.xml /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKTable.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKTableReadOnly.java
          Hide
          hudson Hudson added a comment -

          Integrated in HBase-0.94 #640 (See https://builds.apache.org/job/HBase-0.94/640/)
          HBASE-6775 Use ZK.multi when available (Addendum: add test file) (Revision 1423635)
          HBASE-6775 Use ZK.multi when available for HBASE-6710 0.92/0.94 compatibility fix (Revision 1423616)

          Result = FAILURE
          gchanan :
          Files :

          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKMulti.java

          gchanan :
          Files :

          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/HConstants.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKTable.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
          • /hbase/branches/0.94/src/main/resources/hbase-default.xml
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKTable.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKTableReadOnly.java
          Show
          hudson Hudson added a comment - Integrated in HBase-0.94 #640 (See https://builds.apache.org/job/HBase-0.94/640/ ) HBASE-6775 Use ZK.multi when available (Addendum: add test file) (Revision 1423635) HBASE-6775 Use ZK.multi when available for HBASE-6710 0.92/0.94 compatibility fix (Revision 1423616) Result = FAILURE gchanan : Files : /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKMulti.java gchanan : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/HConstants.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKTable.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java /hbase/branches/0.94/src/main/resources/hbase-default.xml /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKTable.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKTableReadOnly.java
          Hide
          gchanan Gregory Chanan added a comment -

          Should release notes describe this feature ?

          Yes, I'll work on one.

          Show
          gchanan Gregory Chanan added a comment - Should release notes describe this feature ? Yes, I'll work on one.
          Hide
          yuzhihong@gmail.com Ted Yu added a comment -

          Should release notes describe this feature ?

          Show
          yuzhihong@gmail.com Ted Yu added a comment - Should release notes describe this feature ?
          Hide
          gchanan Gregory Chanan added a comment -

          Thanks for the review, Jimmy.

          Committed to 0.94.

          Show
          gchanan Gregory Chanan added a comment - Thanks for the review, Jimmy. Committed to 0.94.
          Hide
          gchanan Gregory Chanan added a comment -

          Alright, going to check in later today if no objections.

          Show
          gchanan Gregory Chanan added a comment - Alright, going to check in later today if no objections.
          Hide
          lhofhansl Lars Hofhansl added a comment -

          My comment from 11/11 should read: we shouldn't sweat a patch. If we have a patch, we can certainly a good improvement for 0.94.

          Show
          lhofhansl Lars Hofhansl added a comment - My comment from 11/11 should read: we shouldn't sweat a patch. If we have a patch, we can certainly a good improvement for 0.94.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12561348/HBASE-6775-v2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 8 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3574//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12561348/HBASE-6775-v2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 8 new or modified tests. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3574//console This message is automatically generated.
          Hide
          gchanan Gregory Chanan added a comment -

          Latest patch from reviewboard.

          Show
          gchanan Gregory Chanan added a comment - Latest patch from reviewboard.
          Hide
          gchanan Gregory Chanan added a comment -

          There's two parts to this patch (perhaps I should have split it initially):
          1. Infrastructure for using ZK.multi. If we want to punt in 0.92/0.94, we may want the infrastructure in 0.96. But perhaps we need a new use case first.
          2. Using the infrastructure for the HBASE-6710 0.92/0.94 compatibility fix. We don't need this part if we punt in 0.92/0.94.

          Show
          gchanan Gregory Chanan added a comment - There's two parts to this patch (perhaps I should have split it initially): 1. Infrastructure for using ZK.multi. If we want to punt in 0.92/0.94, we may want the infrastructure in 0.96. But perhaps we need a new use case first. 2. Using the infrastructure for the HBASE-6710 0.92/0.94 compatibility fix. We don't need this part if we punt in 0.92/0.94.
          Hide
          lhofhansl Lars Hofhansl added a comment -

          I'd be OK with punting on this for 0.92 and 0.94.

          Show
          lhofhansl Lars Hofhansl added a comment - I'd be OK with punting on this for 0.92 and 0.94.
          Hide
          gchanan Gregory Chanan added a comment -

          Woops, wrong copy-paste: https://reviews.apache.org/r/7888/

          Show
          gchanan Gregory Chanan added a comment - Woops, wrong copy-paste: https://reviews.apache.org/r/7888/
          Show
          gchanan Gregory Chanan added a comment - Review Board: https://issues.apache.org/jira/browse/HBASE-6775
          Hide
          stack stack added a comment -

          No objection on my part after the above explanation. Would suggest that you add the above text into the description of the new config.

          Show
          stack stack added a comment - No objection on my part after the above explanation. Would suggest that you add the above text into the description of the new config.
          Hide
          gchanan Gregory Chanan added a comment -

          I think config is the best option. It should only be temporary (gone in 0.96), so it's not making things any more cryptic long term.

          The basic problem is ZOOKEEPER-1495; you can't issue a command and figure out quickly whether the cluster supports multi or not. You can try to figure this out yourself, using "srvr", but it's tricky. You need to check that every node on the cluster supports multi. Apparently just figuring out what servers are in the cluster is complicated (I was told jmx is the best way); the zk configuration in hbase-land is not sufficient because it may not list every node. It is not sufficient to just check for a quorum of servers that support it, because the servers that don't support it will be kicked (remove themselves?) from the quorum and you'll lose your fault tolerance.

          Even if you know every server in the cluster supports multi at some point in time, that may change by the time you actually issue your command. A new server may join or a server version may be downgraded by the time you actually send your command. So there's no good way to do this.

          Any objection to going with the Configuration parameter, Stack or others? (config parameter would be called hbase.zookeeper.useMulti).

          Show
          gchanan Gregory Chanan added a comment - I think config is the best option. It should only be temporary (gone in 0.96), so it's not making things any more cryptic long term. The basic problem is ZOOKEEPER-1495 ; you can't issue a command and figure out quickly whether the cluster supports multi or not. You can try to figure this out yourself, using "srvr", but it's tricky. You need to check that every node on the cluster supports multi. Apparently just figuring out what servers are in the cluster is complicated (I was told jmx is the best way); the zk configuration in hbase-land is not sufficient because it may not list every node. It is not sufficient to just check for a quorum of servers that support it, because the servers that don't support it will be kicked (remove themselves?) from the quorum and you'll lose your fault tolerance. Even if you know every server in the cluster supports multi at some point in time, that may change by the time you actually issue your command. A new server may join or a server version may be downgraded by the time you actually send your command. So there's no good way to do this. Any objection to going with the Configuration parameter, Stack or others? (config parameter would be called hbase.zookeeper.useMulti).
          Hide
          stack stack added a comment -

          Config is a bad idea IMO. We have enough cryptic config. already.

          One thought I had was that you could connect to an ensemble member that supported multi but another member might not. If your session moves, I don't believe the client is told so you could do a multi against an older ensemble member. This might be a bit on the exotic side (different ensemble versions is probably temporary condition while ensemble is being updated)

          Show
          stack stack added a comment - Config is a bad idea IMO. We have enough cryptic config. already. One thought I had was that you could connect to an ensemble member that supported multi but another member might not. If your session moves, I don't believe the client is told so you could do a multi against an older ensemble member. This might be a bit on the exotic side (different ensemble versions is probably temporary condition while ensemble is being updated)
          Hide
          yuzhihong@gmail.com Ted Yu added a comment -

          Finding the version of zookeeper ensemble doesn't have to be achieved by one command.
          If HBase cluster comes up normally but call to multi() times out (more than once), we can be certain that zookeeper ensemble doesn't support multi().
          Giving user config parameter as suggested above is fine too.

          Show
          yuzhihong@gmail.com Ted Yu added a comment - Finding the version of zookeeper ensemble doesn't have to be achieved by one command. If HBase cluster comes up normally but call to multi() times out (more than once), we can be certain that zookeeper ensemble doesn't support multi(). Giving user config parameter as suggested above is fine too.
          Hide
          stack stack added a comment -

          Interested in why its a "bad idea(TM)". Good on you G.

          Show
          stack stack added a comment - Interested in why its a "bad idea(TM)". Good on you G.
          Hide
          gchanan Gregory Chanan added a comment -

          I'm not sure we will require ZK 3.4+ in 0.94.

          Perhaps we could put this behind a Configuration parameter (hbase.zookeeper.serverSupportsMulti or hbase.zookeeper.useMulti), which would give the user the option of using multi if they know their cluster supports it.

          I also contemplated trying to figure this out programmaticaly (calling "srvr" via "send4LetterWord" on the zookeeper quorum), but I've been told this is a bad idea. Trying to get more information on that approach.

          Show
          gchanan Gregory Chanan added a comment - I'm not sure we will require ZK 3.4+ in 0.94. Perhaps we could put this behind a Configuration parameter (hbase.zookeeper.serverSupportsMulti or hbase.zookeeper.useMulti), which would give the user the option of using multi if they know their cluster supports it. I also contemplated trying to figure this out programmaticaly (calling "srvr" via "send4LetterWord" on the zookeeper quorum), but I've been told this is a bad idea. Trying to get more information on that approach.
          Hide
          lhofhansl Lars Hofhansl added a comment -

          No patch... Moving to 0.94.4

          Show
          lhofhansl Lars Hofhansl added a comment - No patch... Moving to 0.94.4
          Hide
          gchanan Gregory Chanan added a comment -

          @Ted:

          Yes, fixed.

          Show
          gchanan Gregory Chanan added a comment - @Ted: Yes, fixed.
          Hide
          yuzhihong@gmail.com Ted Yu added a comment -

          @Gregory:
          Should the Fix Version be 0.94.3 ?

          Thanks

          Show
          yuzhihong@gmail.com Ted Yu added a comment - @Gregory: Should the Fix Version be 0.94.3 ? Thanks

            People

            • Assignee:
              gchanan Gregory Chanan
              Reporter:
              gchanan Gregory Chanan
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development