Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-1392

Improve namenode scalability by prioritizing datanode heartbeats over block reports

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Duplicate
    • None
    • None
    • namenode
    • None

    Description

      When a namenode restarts, it gets heartbeats followed by a block reports from the datanodes. The block report processing is heavyweight and can take many many seconds, especially when the number of blocks in each datanode is large. This causes the namenode to not be able to process heartbeats in time, thus leading to "lost heartbeats from datanode". The namenode declares the datanode to be dead and the datanode has to re-send its register/block report all over again. This is a scalability problem.

      Attachments

        Issue Links

          is blocked by

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-6952 Support sending priority RPC

          • Major - Major loss of function.
          • Open
          is related to

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-599 Improve Namenode robustness by prioritizing datanode heartbeats over client requests

          • Major - Major loss of function.
          • Closed
          relates to

          New Feature - A new feature of the product, which has yet to be developed. HDFS-779 Automatic move to safe-mode when cluster size drops

          • Major - Major loss of function.
          • Open

          Activity

            People

              dhruba Dhruba Borthakur
              dhruba Dhruba Borthakur
              Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: