Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-2126 Improve Namenode startup time [umbrella task]
  3. HDFS-1295

Improve namenode restart times by short-circuiting the first block reports from datanodes

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 0.23.0
    • Federation Branch, 0.23.0
    • namenode
    • None
    • Reviewed

    Description

      The namenode restart is dominated by the performance of processing block reports. On a 2000 node cluster with 90 million blocks, block report processing takes 30 to 40 minutes. The namenode "diffs" the contents of the incoming block report with the contents of the blocks map, and then applies these diffs to the blocksMap, but in reality there is no need to compute the "diff" because this is the first block report from the datanode.

      This code change improves block report processing time by 300%.

      Attachments

        1. HDFS-1295_delta_for_trunk.patch
          2 kB
          Matthew Foley
        2. HDFS-1295_for_ymerge_v2.patch
          33 kB
          Matthew Foley
        3. HDFS-1295_for_ymerge.patch
          32 kB
          Matthew Foley
        4. IBR_shortcut_v2a.patch
          29 kB
          Matthew Foley
        5. IBR_shortcut_v3atrunk.patch
          28 kB
          Matthew Foley
        6. IBR_shortcut_v4atrunk.patch
          49 kB
          Matthew Foley
        7. IBR_shortcut_v4atrunk.patch
          49 kB
          Matthew Foley
        8. IBR_shortcut_v4atrunk.patch
          49 kB
          Matthew Foley
        9. IBR_shortcut_v6atrunk.patch
          49 kB
          Matthew Foley
        10. IBR_shortcut_v7atrunk.patch
          33 kB
          Matthew Foley
        11. shortCircuitBlockReport_1.txt
          10 kB
          Dhruba Borthakur

        Issue Links

          is duplicated by

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-1147 Reduce NN startup time by reducing the processing time of block reports

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              mattf Matthew Foley
              dhruba Dhruba Borthakur
              Votes:
              0 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: