Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-12241

The crash of regionServer when taking deadserver's replication queue breaks replication

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.0, 0.99.2
    • Component/s: Replication
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Hide
      This fix includes our enabling useMulti flag as default. multi is a zk method only available in later versions of zookeeper. This change means HBase 1.0 requires a zookeeper that is at least version 3.4+. See HBASE-6775 for background.
      Show
      This fix includes our enabling useMulti flag as default. multi is a zk method only available in later versions of zookeeper. This change means HBase 1.0 requires a zookeeper that is at least version 3.4+. See HBASE-6775 for background.

      Description

      When a regionserver crash, another regionserver will try to take over the replication hlogs queue and help the the the dead regionserver to finish the replcation.See NodeFailoverWorker in ReplicationSourceManager

      Currently hbase.zookeeper.useMulti is false in default configuration. The operation of taking over replication queue is not atomic. The ReplicationSourceManager firstly lock the replication node of dead regionserver and then copy the replication queue, and delete replication node of dead regionserver at last. The operation of the lockOtherRS just creates a persistent zk node named "lock" which prevent other regionserver taking over the replication queue.
      See:

        public boolean lockOtherRS(String znode) {
          try {
            String parent = ZKUtil.joinZNode(this.rsZNode, znode);
            if (parent.equals(rsServerNameZnode)) {
              LOG.warn("Won't lock because this is us, we're dead!");
              return false;
            }
            String p = ZKUtil.joinZNode(parent, RS_LOCK_ZNODE);
            ZKUtil.createAndWatch(this.zookeeper, p, Bytes.toBytes(rsServerNameZnode));
          } catch (KeeperException e) {
            ...
            return false;
          }
          return true;
        }
      

      But if a regionserver crashed after creating this "lock" zk node and before coping the replication queue to its replication queue, the "lock" zk node will be left forever and
      no other regionserver can take over the replication queue.

      In out production cluster, we encounter this problem. We found the replication queue was there and no regionserver took over it and a "lock" zk node left there.

      hbase.32561.log:2014-09-24,14:09:28,790 INFO org.apache.hadoop.hbase.replication.ReplicationZookeeper: Won't transfer the queue, another RS took care of it because of: KeeperErrorCode = NoNode for /hbase/hhsrv-micloud/replication/rs/hh-hadoop-srv-st09.bj,12610,1410937824255/lock
      hbase.32561.log:2014-09-24,14:14:45,148 INFO org.apache.hadoop.hbase.replication.ReplicationZookeeper: Won't transfer the queue, another RS took care of it because of: KeeperErrorCode = NoNode for /hbase/hhsrv-micloud/replication/rs/hh-hadoop-srv-st10.bj,12600,1410937795685/lock

      A quick solution is that the lock operation just create an ephemeral "lock" zookeeper node and when the lock node is deleted, other regionserver will be notified to check if there are replication queue left.

      Suggestions are welcomed! Thanks.

        Issue Links

          Activity

          Hide
          enis Enis Soztutar added a comment -

          Closing this issue after 0.99.2 release.

          Show
          enis Enis Soztutar added a comment - Closing this issue after 0.99.2 release.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in HBase-TRUNK #5668 (See https://builds.apache.org/job/HBase-TRUNK/5668/)
          HBASE-12241 The crash of regionServer when taking deadserver's replication queue breaks replication (Shaohui) (tedyu: rev 7c87f9c6b58d628d453d3a74ddda18106543cdbf)

          • hbase-common/src/main/resources/hbase-default.xml
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in HBase-TRUNK #5668 (See https://builds.apache.org/job/HBase-TRUNK/5668/ ) HBASE-12241 The crash of regionServer when taking deadserver's replication queue breaks replication (Shaohui) (tedyu: rev 7c87f9c6b58d628d453d3a74ddda18106543cdbf) hbase-common/src/main/resources/hbase-default.xml
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in HBase-1.0 #323 (See https://builds.apache.org/job/HBase-1.0/323/)
          HBASE-12241 The crash of regionServer when taking deadserver's replication queue breaks replication (Shaohui) (tedyu: rev 5b3f6fb1a70133918f6b982b538b4a910aeb5633)

          • hbase-common/src/main/resources/hbase-default.xml
          Show
          hudson Hudson added a comment - FAILURE: Integrated in HBase-1.0 #323 (See https://builds.apache.org/job/HBase-1.0/323/ ) HBASE-12241 The crash of regionServer when taking deadserver's replication queue breaks replication (Shaohui) (tedyu: rev 5b3f6fb1a70133918f6b982b538b4a910aeb5633) hbase-common/src/main/resources/hbase-default.xml
          Hide
          stack stack added a comment -

          Liu Shaohui Don't do a doc patch. There is no place for you to insert such a thing currently. We'll do it as part of the upgrade notes for 1.0 (I added a TODO on HBASE-10856) Thanks.

          Show
          stack stack added a comment - Liu Shaohui Don't do a doc patch. There is no place for you to insert such a thing currently. We'll do it as part of the upgrade notes for 1.0 (I added a TODO on HBASE-10856 ) Thanks.
          Hide
          yuzhihong@gmail.com Ted Yu added a comment -

          Liu Shaohui:
          Can you attach a doc patch ?

          Show
          yuzhihong@gmail.com Ted Yu added a comment - Liu Shaohui : Can you attach a doc patch ?
          Hide
          yuzhihong@gmail.com Ted Yu added a comment -

          Integrated to branch-1 and master

          Thanks for the patch, Shaohui.

          Show
          yuzhihong@gmail.com Ted Yu added a comment - Integrated to branch-1 and master Thanks for the patch, Shaohui.
          Hide
          enis Enis Soztutar added a comment -

          This should come with doc changes about the min zookeeper version of 3.4 though.

          Show
          enis Enis Soztutar added a comment - This should come with doc changes about the min zookeeper version of 3.4 though.
          Hide
          enis Enis Soztutar added a comment -

          We also turn on useMulti by default for some time now. +1 for the branch-1 change. It will help with stability and less configuration to worry.

          Show
          enis Enis Soztutar added a comment - We also turn on useMulti by default for some time now. +1 for the branch-1 change. It will help with stability and less configuration to worry.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12674739/HBASE-12241-trunk-v1.diff
          against trunk revision .
          ATTACHMENT ID: 12674739

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          +1 site. The mvn site goal succeeds with this patch.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/11380//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11380//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11380//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11380//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11380//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11380//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11380//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11380//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11380//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11380//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11380//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11380//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/11380//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12674739/HBASE-12241-trunk-v1.diff against trunk revision . ATTACHMENT ID: 12674739 +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/11380//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11380//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11380//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11380//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11380//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11380//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11380//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11380//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11380//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11380//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11380//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11380//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/11380//console This message is automatically generated.
          Hide
          apurtell Andrew Purtell added a comment -

          +1

          Show
          apurtell Andrew Purtell added a comment - +1
          Hide
          liushaohui Liu Shaohui added a comment -

          Set useMulti to be true in hbase-default.xml

          Show
          liushaohui Liu Shaohui added a comment - Set useMulti to be true in hbase-default.xml
          Hide
          apurtell Andrew Purtell added a comment -

          Let's make useMulti the default as well as bump up the min zookeeper version.

          Show
          apurtell Andrew Purtell added a comment - Let's make useMulti the default as well as bump up the min zookeeper version.
          Hide
          liushaohui Liu Shaohui added a comment -

          Jonathan Hsieh
          Yes, for zk 3.4+, we can set hbase.zookeeper.useMulti = true as default to avoid this problem.
          Should we fix this problem for zk < 3.4?

          Show
          liushaohui Liu Shaohui added a comment - Jonathan Hsieh Yes, for zk 3.4+, we can set hbase.zookeeper.useMulti = true as default to avoid this problem. Should we fix this problem for zk < 3.4?
          Hide
          virag Virag Kothari added a comment -

          I think we should do that. We have also experienced that useMulti is more robust.

          Show
          virag Virag Kothari added a comment - I think we should do that. We have also experienced that useMulti is more robust.
          Hide
          jmhsieh Jonathan Hsieh added a comment -

          Do we think it makes sense to have the min zookeeper version bumped up to something that supports useMulti in the 2.0 branch (zk 3.4+)? what about the 1.0 branch?

          We initially added useMulti support added this 2 years about in HBASE-6775.

          Show
          jmhsieh Jonathan Hsieh added a comment - Do we think it makes sense to have the min zookeeper version bumped up to something that supports useMulti in the 2.0 branch (zk 3.4+)? what about the 1.0 branch? We initially added useMulti support added this 2 years about in HBASE-6775 .
          Hide
          stack stack added a comment -

          Setting fix version so hopefully someone of us will take a look.

          Show
          stack stack added a comment - Setting fix version so hopefully someone of us will take a look.

            People

            • Assignee:
              liushaohui Liu Shaohui
              Reporter:
              liushaohui Liu Shaohui
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development