Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-8389

HBASE-8354 forces Namenode into loop with lease recovery requests

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.94.8
    • Component/s: None
    • Labels:
      None
    • Release Note:
      Hide
      A new config param, hbase.lease.recovery.waiting.period, is introduced for FSHDFSUtils#recoverFileLease().
      It represents the amount of time, in milliseconds, that we wait after issuing recoverLease() request. The default value is 4 seconds.
      Show
      A new config param, hbase.lease.recovery.waiting.period, is introduced for FSHDFSUtils#recoverFileLease(). It represents the amount of time, in milliseconds, that we wait after issuing recoverLease() request. The default value is 4 seconds.

      Description

      We ran hbase 0.94.3 patched with 8354 and observed too many outstanding lease recoveries because of the short retry interval of 1 second between lease recoveries.

      The namenode gets into the following loop:
      1) Receives lease recovery request and initiates recovery choosing a primary datanode every second
      2) A lease recovery is successful and the namenode tries to commit the block under recovery as finalized - this takes < 10 seconds in our environment since we run with tight HDFS socket timeouts.
      3) At step 2), there is a more recent recovery enqueued because of the aggressive retries. This causes the committed block to get preempted and we enter a vicious cycle

      So we do, <initiate_recovery> --> <commit_block> --> <commit_preempted_by_another_recovery>

      This loop is paused after 300 seconds which is the "hbase.lease.recovery.timeout". Hence the MTTR we are observing is 5 minutes which is terrible. Our ZK session timeout is 30 seconds and HDFS stale node detection timeout is 20 seconds.

      Note that before the patch, we do not call recoverLease so aggressively - also it seems that the HDFS namenode is pretty dumb in that it keeps initiating new recoveries for every call. Before the patch, we call recoverLease, assume that the block was recovered, try to get the file, it has zero length since its under recovery, we fail the task and retry until we get a non zero length. So things just work.

      Fixes:
      1) Expecting recovery to occur within 1 second is too aggressive. We need to have a more generous timeout. The timeout needs to be configurable since typically, the recovery takes as much time as the DFS timeouts. The primary datanode doing the recovery tries to reconcile the blocks and hits the timeouts when it tries to contact the dead node. So the recovery is as fast as the HDFS timeouts.

      2) We have another issue I report in HDFS 4721. The Namenode chooses the stale datanode to perform the recovery (since its still alive). Hence the first recovery request is bound to fail. So if we want a tight MTTR, we either need something like HDFS 4721 or we need something like this

      recoverLease(...)
      sleep(1000)
      recoverLease(...)
      sleep(configuredTimeout)
      recoverLease(...)
      sleep(configuredTimeout)

      Where configuredTimeout should be large enough to let the recovery happen but the first timeout is short so that we get past the moot recovery in step #1.

        Attachments

        1. 8389-trunk-v3.txt
          5 kB
          Ted Yu
        2. 8389-0.94-v6.txt
          4 kB
          Ted Yu
        3. 8389-0.94-v5.txt
          1 kB
          Ted Yu
        4. 8389-trunk-v2.txt
          4 kB
          Ted Yu
        5. 8389-0.94-v4.txt
          1 kB
          Ted Yu
        6. 8389-0.94-v3.txt
          1 kB
          Ted Yu
        7. 8389-0.94-v2.txt
          1 kB
          Ted Yu
        8. 8389-0.94.txt
          1 kB
          Ted Yu
        9. 8389-trunk-v2.patch
          2 kB
          Varun Sharma
        10. 8389-trunk-v1.txt
          1 kB
          Ted Yu
        11. sample.patch
          1 kB
          Varun Sharma
        12. nn1.log
          84 kB
          Varun Sharma
        13. nn.log
          121 kB
          Varun Sharma

          Issue Links

            Activity

              People

              • Assignee:
                varunsharma Varun Sharma
                Reporter:
                varunsharma Varun Sharma
              • Votes:
                0 Vote for this issue
                Watchers:
                16 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: