HBase
  1. HBase
  2. HBASE-5099

ZK event thread waiting for root region assignment may block server shutdown handler for the region sever the root region was on

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.92.0, 0.94.0
    • Fix Version/s: 0.92.0, 0.94.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      A RS died. The ServerShutdownHandler kicked in and started the logspliting. SpliLogManager
      installed the tasks asynchronously, then started to wait for them to complete.

      The task znodes were not created actually. The requests were just queued.
      At this time, the zookeeper connection expired. HMaster tried to recover the expired ZK session.
      During the recovery, a new zookeeper connection was created. However, this master became the
      new master again. It tried to assign root and meta.

      Because the dead RS got the old root region, the master needs to wait for the log splitting to complete.
      This waiting holds the zookeeper event thread. So the async create split task is never retried since
      there is only one event thread, which is waiting for the root region assigned.

      1. ZK-event-thread-waiting-for-root.png
        35 kB
        Jimmy Xiang
      2. hbase-5099-v6.patch
        7 kB
        Jimmy Xiang
      3. hbase-5099-v5.patch
        7 kB
        Jimmy Xiang
      4. hbase-5099-v4.patch
        7 kB
        Jimmy Xiang
      5. hbase-5099-v3.patch
        7 kB
        Jimmy Xiang
      6. hbase-5099-v2.patch
        6 kB
        Jimmy Xiang
      7. hbase-5099.patch
        6 kB
        Jimmy Xiang
      8. distributed-log-splitting-hangs.png
        42 kB
        Jimmy Xiang
      9. 5099.92
        9 kB
        Ted Yu

        Issue Links

          Activity

          Jimmy Xiang created issue -
          Jimmy Xiang made changes -
          Field Original Value New Value
          Attachment distributed-log-splitting-hangs.png [ 12508707 ]
          Attachment ZK-event-thread-waiting-for-root.png [ 12508708 ]
          Jimmy Xiang made changes -
          Assignee Jimmy Xiang [ jxiang ]
          Jimmy Xiang made changes -
          Attachment hbase-5099.patch [ 12508783 ]
          Jimmy Xiang made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Jimmy Xiang made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Jimmy Xiang made changes -
          Attachment hbase-5099-v2.patch [ 12508799 ]
          Jimmy Xiang made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Jimmy Xiang made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Jimmy Xiang made changes -
          Attachment hbase-5099-v3.patch [ 12508868 ]
          Jimmy Xiang made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Jimmy Xiang made changes -
          Attachment hbase-5099-v4.patch [ 12508887 ]
          Jimmy Xiang made changes -
          Attachment hbase-5099-v5.patch [ 12508888 ]
          Ted Yu made changes -
          Attachment 5099.92 [ 12508891 ]
          Ted Yu made changes -
          Hadoop Flags Reviewed [ 10343 ]
          Fix Version/s 0.92.0 [ 12314223 ]
          Fix Version/s 0.94.0 [ 12316419 ]
          Jimmy Xiang made changes -
          Attachment hbase-5099-v6.patch [ 12508893 ]
          Ted Yu made changes -
          Attachment 5099.92 [ 12508891 ]
          Ted Yu made changes -
          Attachment 5099.92 [ 12508896 ]
          Ted Yu made changes -
          Summary ZK event thread waiting for root region while server shutdown handler waiting for event thread to finish distributed log splitting to recover the region sever the root region is on ZK event thread waiting for root region assignment may block server shutdown handler for the region sever the root region was on
          Ted Yu made changes -
          Comment [ -1 overall. Here are the results of testing the latest attachment
            http://issues.apache.org/jira/secure/attachment/12508896/5099.92
            against trunk revision .

              +1 @author. The patch does not contain any @author tags.

              +1 tests included. The patch appears to include 3 new or modified tests.

              -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/636//console

          This message is automatically generated. ]
          Jimmy Xiang made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Ted Yu made changes -
          Resolution Fixed [ 1 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          Jimmy Xiang made changes -
          Link This issue is related to HBASE-5112 [ HBASE-5112 ]
          Jimmy Xiang made changes -
          Status Reopened [ 4 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Lars Hofhansl made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Jimmy Xiang
              Reporter:
              Jimmy Xiang
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development