Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-20634

Reopen region while server crash can cause the procedure to be stuck

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • None
    • 3.0.0-alpha-1, 2.1.0, 2.0.1
    • None
    • None
    • Reviewed
    • Hide
      A second attempt at fixing HBASE-20173. Fixes unfinished keeping of server state inside AM (ONLINE=>SPLITTING=>OFFLINE=>null). Concurrent unassigns look at server state to figure if they should wait on SCP to wake them up or not.
      Show
      A second attempt at fixing HBASE-20173 . Fixes unfinished keeping of server state inside AM (ONLINE=>SPLITTING=>OFFLINE=>null). Concurrent unassigns look at server state to figure if they should wait on SCP to wake them up or not.

    Description

      Found this when implementing HBASE-20424, where we will transit the peer sync replication state while there is server crash.

      The problem is that, in ServerCrashAssign, we do not have the region lock, so it is possible that after we call handleRIT to clear the existing assign/unassign procedures related to this rs, and before we schedule the assign procedures, it is possible that that we schedule a unassign procedure for a region on the crashed rs. This procedure will not receive the ServerCrashException, instead, in addToRemoteDispatcher, it will find that it can not dispatch the remote call and then a FailedRemoteDispatchException will be raised. But we do not treat this exception the same with ServerCrashException, instead, we will try to expire the rs. Obviously the rs has already been marked as expired, so this is almost a no-op. Then the procedure will be stuck there for ever.

      A possible way to fix it is to treat FailedRemoteDispatchException the same with ServerCrashException, as it will be created in addToRemoteDispatcher only, and the only reason we can not dispatch a remote call is that the rs has already been dead. The nodeMap is a ConcurrentMap so I think we could use it as a guard.

      Attachments

        1. HBASE-20634.branch-2.0.001.patch
          41 kB
          Michael Stack
        2. HBASE-20634.branch-2.0.002.patch
          41 kB
          Michael Stack
        3. HBASE-20634.branch-2.0.003.patch
          41 kB
          Michael Stack
        4. HBASE-20634.branch-2.0.004.patch
          41 kB
          Michael Stack
        5. HBASE-20634.branch-2.0.005.patch
          43 kB
          Michael Stack
        6. HBASE-20634.branch-2.0.006.patch
          43 kB
          Michael Stack
        7. HBASE-20634.branch-2.0.006.patch
          43 kB
          Michael Stack
        8. HBASE-20634.branch-2.0.007.patch
          45 kB
          Michael Stack
        9. HBASE-20634.branch-2.0.008.patch
          46 kB
          Michael Stack
        10. HBASE-20634.branch-2.0.009.patch
          46 kB
          Michael Stack
        11. HBASE-20634-UT.patch
          7 kB
          Duo Zhang

        Issue Links

          relates to

          Sub-task - The sub-task of the issue HBASE-20173 [AMv2] DisableTableProcedure concurrent to ServerCrashProcedure can deadlock

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          links to

          Web Link Review Board (branch-2.0)

          Activity

            People

              stack Michael Stack
              zhangduo Duo Zhang
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: