Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-4183

Throttle block recovery

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Critical
    • Resolution: Duplicate
    • Affects Version/s: 0.23.4, 2.0.2-alpha
    • Fix Version/s: None
    • Component/s: namenode
    • Labels:
      None

      Description

      When a large number of files are abandoned without closing, a storm of lease expiration follows in about an hour (lease hard limit). For the last block of each file, block recovery is initiated and when the datanode is done, it calls commitBlockSynchronization() is called against namenode. A burst of these calls can slow down namenode considerably. We need to throttle block recovery and/or speed up the rate at which commitBlockSynchronization() is served.

        Attachments

        Issue Links

          Activity

            People

            • Assignee:
              kihwal Kihwal Lee
              Reporter:
              kihwal Kihwal Lee

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment