Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-1392

Improve namenode scalability by prioritizing datanode heartbeats over block reports

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Duplicate
    • None
    • None
    • namenode
    • None

    Description

      When a namenode restarts, it gets heartbeats followed by a block reports from the datanodes. The block report processing is heavyweight and can take many many seconds, especially when the number of blocks in each datanode is large. This causes the namenode to not be able to process heartbeats in time, thus leading to "lost heartbeats from datanode". The namenode declares the datanode to be dead and the datanode has to re-send its register/block report all over again. This is a scalability problem.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            dhruba Dhruba Borthakur
            dhruba Dhruba Borthakur
            Votes:
            0 Vote for this issue
            Watchers:
            13 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment