HBase
  1. HBase
  2. HBASE-8223

[89-FB] Fix race condition between append/sync and HLog.close() work flow.

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: 0.89-fb
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      The RegionServer was performing a shutdown as it got YouAreDeadException. It closed the HLog and was waiting to close all the regions where it got stuck.

      The reason region close is stuck because it's trying to obtain a lock which is occupied by the put op, which in turn is waiting for append to HLog to complete.

      There is race condition here is between append/sync and HLog.close() work flow.

      Scenario:

      Thread 1 => doing the append
      Thread 2 => doing HregionServer shutdown

      Timeline:

      t1> 1: Verifies that LogSyncer is not shutting down and HLog is not closed and calls sync()
      t2> 2: HRegionServer issued HLog.close()
      t3> 2: In HLog.close(), it joins the LogSyncer thread, which signals all the threads waiting on syncDone and exits.
      t4> 1: In sync, it sees that the sync has not complete until its txd, hence adds itself to the syncDone.await queue.

      Note: at t4, it does not check whether the LogSyncer Thread is alive or not, which caused this hang.

        Activity

        Rishit Shroff created issue -
        Rishit Shroff made changes -
        Field Original Value New Value
        Assignee Rishit Shroff [ rshroff ]

          People

          • Assignee:
            Rishit Shroff
            Reporter:
            Rishit Shroff
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:

              Development