HBase
  1. HBase
  2. HBASE-6710

0.92/0.94 compatibility issues due to HBASE-5206

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.94.2
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      This issue introduces a compatibility mode on the HMaster for 0.92.0 and 0.92.1 clients. Without the compatibility mode, 0.92.0 and 0.92.1 clients will hang on calls to "enableTable" and "is_enabled" will always return false, even for enabled tables. To use the compatibility mode, 0.92.0 and 0.92.1 clients require a restart with the following configuration change:
      <name>zookeeper.znode.tableEnableDisable</name>
      <value>table92</value>
      In rare failure cases, even with the compatibility mode on, the client may report incorrect results for "is_enabled" and "is_disabled." For example, "is_enabled" may return true even though the table is disabled (the correct value can be checked via the HMaster UI). This issue can be corrected by calling "enable" or "disable" to return the table to the desired state.
      Show
      This issue introduces a compatibility mode on the HMaster for 0.92.0 and 0.92.1 clients. Without the compatibility mode, 0.92.0 and 0.92.1 clients will hang on calls to "enableTable" and "is_enabled" will always return false, even for enabled tables. To use the compatibility mode, 0.92.0 and 0.92.1 clients require a restart with the following configuration change: <name>zookeeper.znode.tableEnableDisable</name> <value>table92</value> In rare failure cases, even with the compatibility mode on, the client may report incorrect results for "is_enabled" and "is_disabled." For example, "is_enabled" may return true even though the table is disabled (the correct value can be checked via the HMaster UI). This issue can be corrected by calling "enable" or "disable" to return the table to the desired state.

      Description

      HBASE-5206 introduces some compatibility issues between

      {0.94,0.94.1}

      and

      {0.92.0,0.92.1}. The release notes of HBASE-5155 describes the issue (HBASE-5206 is a backport of HBASE-5155).

      I think we can make 0.94.2 compatible with both {0.94.0,0.94.1} and {0.92.0,0.92.1}

      , although one of those sets will require configuration changes.

      The basic problem is that there is a znode for each table "zookeeper.znode.tableEnableDisable" that is handled differently.

      On 0.92.0 and 0.92.1 the states for this table are:
      [ disabled, disabling, enabling ] or deleted if the table is enabled

      On 0.94.1 and 0.94.2 the states for this table are:
      [ disabled, disabling, enabling, enabled ]

      What saves us is that the location of this znode is configurable. So the basic idea is to have the 0.94.2 master write two different znodes, "zookeeper.znode.tableEnableDisabled92" and "zookeeper.znode.tableEnableDisabled94" where the 92 node is in 92 format, the 94 node is in 94 format. And internally, the master would only use the 94 format in order to solve the original bug HBASE-5155 solves.
      We can of course make one of these the same default as exists now, so we don't need to make config changes for one of 0.92 or 0.94 clients. I argue that 0.92 clients shouldn't have to make config changes for the same reason I argued above. But that is debatable.

      Then, I think the only question left is the question of how to bring along the

      {0.94.0, 0.94.1} crew. A {0.94.0, 0.94.1}

      client would work against a 0.94.2 cluster by just configuring "zookeeper.znode.tableEnableDisable" in the client to be whatever "zookeeper.znode.tableEnableDisabled94" is in the cluster. A 0.94.2 client would work against both a

      {0.94.0, 0.94.1} and {0.92.0, 0.92.1} cluster if it had HBASE-6268 applied. About rolling upgrade from {0.94.0, 0.94.1}

      to 0.94.2 – I'd have to think about that. Do the regionservers ever read the tableEnableDisabled znode?

      On the mailing list, Lars H suggested the following:
      "The only input I'd have is that format we'll use going forward will not have a version attached to it.
      So maybe the 92 version would still be called "zookeeper.znode.tableEnableDisable" and the new node could have a different name "zookeeper.znode.tableEnableDisableNew" (or something)."

      1. HBASE-6710-v3.patch
        35 kB
        Gregory Chanan

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in HBase-0.94-security-on-Hadoop-23 #10 (See https://builds.apache.org/job/HBase-0.94-security-on-Hadoop-23/10/)
          HBASE-6775 Use ZK.multi when available for HBASE-6710 0.92/0.94 compatibility fix (Revision 1423616)

          Result = FAILURE
          gchanan :
          Files :

          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/HConstants.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKTable.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
          • /hbase/branches/0.94/src/main/resources/hbase-default.xml
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKTable.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKTableReadOnly.java
          Show
          Hudson added a comment - Integrated in HBase-0.94-security-on-Hadoop-23 #10 (See https://builds.apache.org/job/HBase-0.94-security-on-Hadoop-23/10/ ) HBASE-6775 Use ZK.multi when available for HBASE-6710 0.92/0.94 compatibility fix (Revision 1423616) Result = FAILURE gchanan : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/HConstants.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKTable.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java /hbase/branches/0.94/src/main/resources/hbase-default.xml /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKTable.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKTableReadOnly.java
          Hide
          Hudson added a comment -

          Integrated in HBase-0.94-security #87 (See https://builds.apache.org/job/HBase-0.94-security/87/)
          HBASE-6775 Use ZK.multi when available for HBASE-6710 0.92/0.94 compatibility fix (Revision 1423616)

          Result = SUCCESS
          gchanan :
          Files :

          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/HConstants.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKTable.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
          • /hbase/branches/0.94/src/main/resources/hbase-default.xml
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKTable.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKTableReadOnly.java
          Show
          Hudson added a comment - Integrated in HBase-0.94-security #87 (See https://builds.apache.org/job/HBase-0.94-security/87/ ) HBASE-6775 Use ZK.multi when available for HBASE-6710 0.92/0.94 compatibility fix (Revision 1423616) Result = SUCCESS gchanan : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/HConstants.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKTable.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java /hbase/branches/0.94/src/main/resources/hbase-default.xml /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKTable.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKTableReadOnly.java
          Hide
          Hudson added a comment -

          Integrated in HBase-0.94 #640 (See https://builds.apache.org/job/HBase-0.94/640/)
          HBASE-6775 Use ZK.multi when available for HBASE-6710 0.92/0.94 compatibility fix (Revision 1423616)

          Result = FAILURE
          gchanan :
          Files :

          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/HConstants.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKTable.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
          • /hbase/branches/0.94/src/main/resources/hbase-default.xml
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKTable.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKTableReadOnly.java
          Show
          Hudson added a comment - Integrated in HBase-0.94 #640 (See https://builds.apache.org/job/HBase-0.94/640/ ) HBASE-6775 Use ZK.multi when available for HBASE-6710 0.92/0.94 compatibility fix (Revision 1423616) Result = FAILURE gchanan : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/HConstants.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKTable.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java /hbase/branches/0.94/src/main/resources/hbase-default.xml /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKTable.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKTableReadOnly.java
          Hide
          Hudson added a comment -

          Integrated in HBase-0.94-security-on-Hadoop-23 #8 (See https://builds.apache.org/job/HBase-0.94-security-on-Hadoop-23/8/)
          HBASE-6710 0.92/0.94 compatibility issues due to HBASE-5206 (Revision 1384181)

          Result = FAILURE
          gchanan :
          Files :

          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKTable.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKTableReadOnly.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKTable.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKTableReadOnly.java
          Show
          Hudson added a comment - Integrated in HBase-0.94-security-on-Hadoop-23 #8 (See https://builds.apache.org/job/HBase-0.94-security-on-Hadoop-23/8/ ) HBASE-6710 0.92/0.94 compatibility issues due to HBASE-5206 (Revision 1384181) Result = FAILURE gchanan : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKTable.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKTableReadOnly.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKTable.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKTableReadOnly.java
          Hide
          Gregory Chanan added a comment -

          Added Release Note

          Show
          Gregory Chanan added a comment - Added Release Note
          Hide
          ramkrishna.s.vasudevan added a comment -

          So as per the change the 0.92 clients need a restart with configuration changes and not the 0.94 client. Thanks Gregory.

          Show
          ramkrishna.s.vasudevan added a comment - So as per the change the 0.92 clients need a restart with configuration changes and not the 0.94 client. Thanks Gregory.
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #175 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/175/)
          HBASE-6759 Forward port ZKReadOnly change from HBASE-6710 (Revision 1384987)

          Result = FAILURE
          gchanan :
          Files :

          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKTable.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKTableReadOnly.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #175 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/175/ ) HBASE-6759 Forward port ZKReadOnly change from HBASE-6710 (Revision 1384987) Result = FAILURE gchanan : Files : /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKTable.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKTableReadOnly.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java
          Hide
          Lars Hofhansl added a comment -

          Last release notes look good. +1

          Show
          Lars Hofhansl added a comment - Last release notes look good. +1
          Hide
          stack added a comment -

          +1 on release note.

          You might rephrase the first few sentences as below but not important.

          "This issue introduces a compatibility mode on the HMaster for 0.92.0 and 0.92.1 clients. Without the compatibility mode..."

          Show
          stack added a comment - +1 on release note. You might rephrase the first few sentences as below but not important. "This issue introduces a compatibility mode on the HMaster for 0.92.0 and 0.92.1 clients. Without the compatibility mode..."
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK #3334 (See https://builds.apache.org/job/HBase-TRUNK/3334/)
          HBASE-6759 Forward port ZKReadOnly change from HBASE-6710 (Revision 1384987)

          Result = FAILURE
          gchanan :
          Files :

          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKTable.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKTableReadOnly.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK #3334 (See https://builds.apache.org/job/HBase-TRUNK/3334/ ) HBASE-6759 Forward port ZKReadOnly change from HBASE-6710 (Revision 1384987) Result = FAILURE gchanan : Files : /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKTable.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKTableReadOnly.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java
          Hide
          Gregory Chanan added a comment -

          Here's a second draft of the release note...

          This issue introduces a compatibility mode on the HMaster for 0.92.0 and 0.92.1 clients that requires a configuration change on those client to turn on. Without the compatibility mode, 0.92.0 and 0.92.1 clients will hang on calls to "enableTable" and "is_enabled" will always return false, even for enabled tables. To use the compatibility mode, 0.92.0 and 0.92.1 clients should make the following configuration change:
          <name>zookeeper.znode.tableEnableDisable</name>
          <value>table92</value>
          In rare failure cases, even with the compatibility mode on, the client may report incorrect results for "is_enabled" and "is_disabled." For example, "is_enabled" may return true even though the table is disabled (the correct value can be checked via the HMaster UI). This issue can be corrected by calling "enable" or "disable" to return the table to the desired state.

          Show
          Gregory Chanan added a comment - Here's a second draft of the release note... This issue introduces a compatibility mode on the HMaster for 0.92.0 and 0.92.1 clients that requires a configuration change on those client to turn on. Without the compatibility mode, 0.92.0 and 0.92.1 clients will hang on calls to "enableTable" and "is_enabled" will always return false, even for enabled tables. To use the compatibility mode, 0.92.0 and 0.92.1 clients should make the following configuration change: <name>zookeeper.znode.tableEnableDisable</name> <value>table92</value> In rare failure cases, even with the compatibility mode on, the client may report incorrect results for "is_enabled" and "is_disabled." For example, "is_enabled" may return true even though the table is disabled (the correct value can be checked via the HMaster UI). This issue can be corrected by calling "enable" or "disable" to return the table to the desired state.
          Hide
          Gregory Chanan added a comment -

          Should we fail silently if node deletion fails ? Would this increase the chance of inconsistency between znode92 and znode ?

          The name "deleteNodeFailSilent" is bad. It's only silent on a NoNodeException, which is fine because that's the state you want anyway.

          Show
          Gregory Chanan added a comment - Should we fail silently if node deletion fails ? Would this increase the chance of inconsistency between znode92 and znode ? The name "deleteNodeFailSilent" is bad. It's only silent on a NoNodeException, which is fine because that's the state you want anyway.
          Hide
          Gregory Chanan added a comment -

          I don't expect user to issue 'enable' command just to verify whether the table is really disabled

          Right, but what they could do, is call "disable" which blocks until "is_disabled" returns true. "is_disabled" may return true incorrectly and a subsequent call to "enable" may throw a TableNotDisabledException.

          Show
          Gregory Chanan added a comment - I don't expect user to issue 'enable' command just to verify whether the table is really disabled Right, but what they could do, is call "disable" which blocks until "is_disabled" returns true. "is_disabled" may return true incorrectly and a subsequent call to "enable" may throw a TableNotDisabledException.
          Hide
          Gregory Chanan added a comment -

          Should hbck be enhanced to detect / fix such inconsistencies ?

          It could, but I don't think it's worth the effort. You only see the issue on 0.92.0/0.92.1 clients, and hbck in those versions can't fix it anyway. So you'd have to go to a 0.94.2 client and run hbck, when you could just retry on the 0.92.0/0.92.1 client.

          Show
          Gregory Chanan added a comment - Should hbck be enhanced to detect / fix such inconsistencies ? It could, but I don't think it's worth the effort. You only see the issue on 0.92.0/0.92.1 clients, and hbck in those versions can't fix it anyway. So you'd have to go to a 0.94.2 client and run hbck, when you could just retry on the 0.92.0/0.92.1 client.
          Hide
          Ted Yu added a comment -

          I don't expect user to issue 'enable' command just to verify whether the table is really disabled

          Show
          Ted Yu added a comment - I don't expect user to issue 'enable' command just to verify whether the table is really disabled
          Hide
          Gregory Chanan added a comment -

          Above comment should read "enable" not "enabled"

          Show
          Gregory Chanan added a comment - Above comment should read "enable" not "enabled"
          Hide
          Gregory Chanan added a comment -

          How does the user verify whether the above commands returned inaccurate results?

          Operations on the table will behave as if the table is not actually in the state returned by "is_enabled" or "is_disabled". For example, "is_disabled" may return true, but a subsequent call to "enabled" may throw a TableNotDisabledException.

          Show
          Gregory Chanan added a comment - How does the user verify whether the above commands returned inaccurate results? Operations on the table will behave as if the table is not actually in the state returned by "is_enabled" or "is_disabled". For example, "is_disabled" may return true, but a subsequent call to "enabled" may throw a TableNotDisabledException.
          Hide
          Ted Yu added a comment -

          I was trying to wrap up 0.92.2 release. So pardon me for late comments.

          Looking at code for void setTableState(final String tableName, final TableState state):

                if (settingToEnabled) {
                  ZKUtil.deleteNodeFailSilent(this.watcher, znode92);
                }
          

          Should we fail silently if node deletion fails ? Would this increase the chance of inconsistency between znode92 and znode ?

          Should hbck be enhanced to detect / fix such inconsistencies ?

          Show
          Ted Yu added a comment - I was trying to wrap up 0.92.2 release. So pardon me for late comments. Looking at code for void setTableState(final String tableName, final TableState state): if (settingToEnabled) { ZKUtil.deleteNodeFailSilent( this .watcher, znode92); } Should we fail silently if node deletion fails ? Would this increase the chance of inconsistency between znode92 and znode ? Should hbck be enhanced to detect / fix such inconsistencies ?
          Hide
          Ted Yu added a comment -

          the client may report inaccurate results for "is_enabled" and "is_disabled."

          How does user verify whether the above commands returned inaccurate results ?

          Thanks

          Show
          Ted Yu added a comment - the client may report inaccurate results for "is_enabled" and "is_disabled." How does user verify whether the above commands returned inaccurate results ? Thanks
          Hide
          Gregory Chanan added a comment -

          Here's a first cut. Let me know what you think.

          This issue introduces a compatibility mode on the HMaster for 0.92.0 and 0.92.1 clients that requires a configuration change on those client to turn on. Without the compatibility mode, 0.92.0 and 0.92.1 clients will hang on calls to "enableTable" and "is_enabled" will always return false, even for enabled tables. To use the compatibility mode, 0.92.0 and 0.92.1 clients should make the following configuration change:
          <name>zookeeper.znode.tableEnableDisable</name>
          <value>table92</value>

          In rare failure cases, even with the compatibility mode on, the client may report inaccurate results for "is_enabled" and "is_disabled." This issue can be corrected by calling "enable" or "disable" again to return the table to the desired state.

          Show
          Gregory Chanan added a comment - Here's a first cut. Let me know what you think. This issue introduces a compatibility mode on the HMaster for 0.92.0 and 0.92.1 clients that requires a configuration change on those client to turn on. Without the compatibility mode, 0.92.0 and 0.92.1 clients will hang on calls to "enableTable" and "is_enabled" will always return false, even for enabled tables. To use the compatibility mode, 0.92.0 and 0.92.1 clients should make the following configuration change: <name>zookeeper.znode.tableEnableDisable</name> <value>table92</value> In rare failure cases, even with the compatibility mode on, the client may report inaccurate results for "is_enabled" and "is_disabled." This issue can be corrected by calling "enable" or "disable" again to return the table to the desired state.
          Hide
          Gregory Chanan added a comment -

          Yes, I'll work on a Release Note.

          Show
          Gregory Chanan added a comment - Yes, I'll work on a Release Note.
          Hide
          Lars Hofhansl added a comment -

          Agreed.
          @Gregory: Wanna take a stab at it. I'm not sure I would do it justice

          Show
          Lars Hofhansl added a comment - Agreed. @Gregory: Wanna take a stab at it. I'm not sure I would do it justice
          Hide
          Jonathan Hsieh added a comment -

          +1 for ted's suggestion.

          Show
          Jonathan Hsieh added a comment - +1 for ted's suggestion.
          Hide
          Ted Yu added a comment -

          Should this JIRA have a detailed Release Notes ?

          Show
          Ted Yu added a comment - Should this JIRA have a detailed Release Notes ?
          Hide
          Hudson added a comment -

          Integrated in HBase-0.94-security #52 (See https://builds.apache.org/job/HBase-0.94-security/52/)
          HBASE-6710 0.92/0.94 compatibility issues due to HBASE-5206 (Revision 1384181)

          Result = SUCCESS
          gchanan :
          Files :

          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKTable.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKTableReadOnly.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKTable.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKTableReadOnly.java
          Show
          Hudson added a comment - Integrated in HBase-0.94-security #52 (See https://builds.apache.org/job/HBase-0.94-security/52/ ) HBASE-6710 0.92/0.94 compatibility issues due to HBASE-5206 (Revision 1384181) Result = SUCCESS gchanan : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKTable.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKTableReadOnly.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKTable.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKTableReadOnly.java
          Hide
          Hudson added a comment -

          Integrated in HBase-0.94 #465 (See https://builds.apache.org/job/HBase-0.94/465/)
          HBASE-6710 0.92/0.94 compatibility issues due to HBASE-5206 (Revision 1384181)

          Result = SUCCESS
          gchanan :
          Files :

          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKTable.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKTableReadOnly.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKTable.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKTableReadOnly.java
          Show
          Hudson added a comment - Integrated in HBase-0.94 #465 (See https://builds.apache.org/job/HBase-0.94/465/ ) HBASE-6710 0.92/0.94 compatibility issues due to HBASE-5206 (Revision 1384181) Result = SUCCESS gchanan : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKTable.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKTableReadOnly.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKTable.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKTableReadOnly.java
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12544917/HBASE-6710-v3.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 8 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2853//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12544917/HBASE-6710-v3.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 8 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2853//console This message is automatically generated.
          Hide
          Gregory Chanan added a comment -

          Committed to 0.94.

          Thanks for the review, Lars.

          Show
          Gregory Chanan added a comment - Committed to 0.94. Thanks for the review, Lars.
          Hide
          Lars Hofhansl added a comment -

          +1 for last patch on RB (v3). This is a great patch with thoughtful testing. Thanks for working this out Gregory.

          Show
          Lars Hofhansl added a comment - +1 for last patch on RB (v3). This is a great patch with thoughtful testing. Thanks for working this out Gregory.
          Hide
          Lars Hofhansl added a comment -

          Some more comments on RB... Let's get this in, so that I can spin 0.94.2RC0

          Show
          Lars Hofhansl added a comment - Some more comments on RB... Let's get this in, so that I can spin 0.94.2RC0
          Hide
          Lars Hofhansl added a comment -

          I commented on RB

          Show
          Lars Hofhansl added a comment - I commented on RB
          Hide
          Ted Yu added a comment -

          Here is the link to review request: https://reviews.apache.org/r/6934

          Show
          Ted Yu added a comment - Here is the link to review request: https://reviews.apache.org/r/6934
          Hide
          Gregory Chanan added a comment -

          @Lars:

          I'm going to post the reviewboard tonight.

          Show
          Gregory Chanan added a comment - @Lars: I'm going to post the reviewboard tonight.
          Hide
          Lars Hofhansl added a comment -

          I'll be out for a few days.

          @Gregory: This is the last issue before 0.94.2. If you get a chance to work on it that'd awesome.

          Show
          Lars Hofhansl added a comment - I'll be out for a few days. @Gregory: This is the last issue before 0.94.2. If you get a chance to work on it that'd awesome.
          Hide
          Lars Hofhansl added a comment -

          After we make this change it'll mean a client restart is required for 0.94.0 and 0.94.1 clients only.

          Show
          Lars Hofhansl added a comment - After we make this change it'll mean a client restart is required for 0.94.0 and 0.94.1 clients only.
          Hide
          ramkrishna.s.vasudevan added a comment -

          Just to clarify
          By meaning new format and old format we mean that if znode is not available after ENABLED then it is old and if the znode is available after ENABLED then it is new format right?
          I can understand how the master will make use of the configuration. So for the client we need to add the 'zookeeper.znode.tableEnableDisable' configuration right? So will this may need a client restart right?

          Show
          ramkrishna.s.vasudevan added a comment - Just to clarify By meaning new format and old format we mean that if znode is not available after ENABLED then it is old and if the znode is available after ENABLED then it is new format right? I can understand how the master will make use of the configuration. So for the client we need to add the 'zookeeper.znode.tableEnableDisable' configuration right? So will this may need a client restart right?
          Hide
          Lars Hofhansl added a comment -

          Interesting. I think it would be awash either way.

          I have a slight preference for using the new znode all over where possible. Just feels right

          Either way this fine. In 0.96 we break everything so, we'll retire the old znode then.

          Show
          Lars Hofhansl added a comment - Interesting. I think it would be awash either way. I have a slight preference for using the new znode all over where possible. Just feels right Either way this fine. In 0.96 we break everything so, we'll retire the old znode then.
          Hide
          Jonathan Hsieh added a comment -

          A simple test in my mind is for lars's point #3 – if you write instructions on how to upgrade in the future, assuming you have the same configuration files, which would be shorter? Ideally we'd choose as the default the approach that ends up being easier.

          Show
          Jonathan Hsieh added a comment - A simple test in my mind is for lars's point #3 – if you write instructions on how to upgrade in the future, assuming you have the same configuration files, which would be shorter? Ideally we'd choose as the default the approach that ends up being easier.
          Hide
          Gregory Chanan added a comment -

          @Lars: yes I plan to work on this, I have time this week.

          Only one issue with your 5 point summary. On point 3, since new clients can deal with both formats (point 1), I figured they might as well use the current znode.

          Show
          Gregory Chanan added a comment - @Lars: yes I plan to work on this, I have time this week. Only one issue with your 5 point summary. On point 3, since new clients can deal with both formats (point 1), I figured they might as well use the current znode.
          Hide
          Lars Hofhansl added a comment -

          @Gregory: I assume you plan to work on this...?
          (Just making sure we're not assuming that the other will work on this)

          Does my 5 point summary above make sense? This is the last issue before I'd like to cut the first 0.94.2 RC.

          Show
          Lars Hofhansl added a comment - @Gregory: I assume you plan to work on this...? (Just making sure we're not assuming that the other will work on this) Does my 5 point summary above make sense? This is the last issue before I'd like to cut the first 0.94.2 RC.
          Hide
          Lars Hofhansl added a comment -

          So to be specific:

          1. With HBASE-6268 in place a client (0.92.2+, 0.94.2+) can deal with both server formats.
          2. With this with jira we'll write the old format in addition to the new format. The old format is written to the current znode. The new format goes into a new znode.
          3. The new servers and new client will use this new znode by default.
          4. Old 0.92 clients (0.92.0, 0.92.1) will use the old format in the current znode
          5. Old 0.94 clients (0.94.0, 0.94.1) need a config change to be pointed to the new znode (because that is the format they understand).

          I do not think we need to make the server side change in 0.92.2. Folks who want a mixed environment should upgrade to 0.94.2. Also if somebody wants to upgrade 0.92 to 0.94, 0.94.2 would be the path of choice (once available).

          Show
          Lars Hofhansl added a comment - So to be specific: With HBASE-6268 in place a client (0.92.2+, 0.94.2+) can deal with both server formats. With this with jira we'll write the old format in addition to the new format. The old format is written to the current znode. The new format goes into a new znode. The new servers and new client will use this new znode by default. Old 0.92 clients (0.92.0, 0.92.1) will use the old format in the current znode Old 0.94 clients (0.94.0, 0.94.1) need a config change to be pointed to the new znode (because that is the format they understand). I do not think we need to make the server side change in 0.92.2. Folks who want a mixed environment should upgrade to 0.94.2. Also if somebody wants to upgrade 0.92 to 0.94, 0.94.2 would be the path of choice (once available).
          Hide
          stack added a comment -

          Do the regionservers ever read the tableEnableDisabled znode?

          I believe this a master-only facility used by master assigning regions (ZKTable is referred to by AM, HMaster, and handlers that run in the master only except in HCM, where we'll check isEnableTable, isDisableTable.... If I grep isEnabledTable/isDisabledTable under regionserver, I don't see references).

          Show
          stack added a comment - Do the regionservers ever read the tableEnableDisabled znode? I believe this a master-only facility used by master assigning regions (ZKTable is referred to by AM, HMaster, and handlers that run in the master only except in HCM, where we'll check isEnableTable, isDisableTable.... If I grep isEnabledTable/isDisabledTable under regionserver, I don't see references).
          Hide
          Gregory Chanan added a comment -

          We could also do the same thing for 0.92.2 if we want 0.92.2 servers to be compatible with 0.94.1 and 0.94.2 clients.

          Show
          Gregory Chanan added a comment - We could also do the same thing for 0.92.2 if we want 0.92.2 servers to be compatible with 0.94.1 and 0.94.2 clients.

            People

            • Assignee:
              Gregory Chanan
              Reporter:
              Gregory Chanan
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development