Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-2126 Improve Namenode startup time [umbrella task]
  3. HDFS-1295

Improve namenode restart times by short-circuiting the first block reports from datanodes

    Details

    • Type: Sub-task Sub-task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: Federation Branch, 0.23.0
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The namenode restart is dominated by the performance of processing block reports. On a 2000 node cluster with 90 million blocks, block report processing takes 30 to 40 minutes. The namenode "diffs" the contents of the incoming block report with the contents of the blocks map, and then applies these diffs to the blocksMap, but in reality there is no need to compute the "diff" because this is the first block report from the datanode.

      This code change improves block report processing time by 300%.

      1. shortCircuitBlockReport_1.txt
        10 kB
        dhruba borthakur
      2. IBR_shortcut_v7atrunk.patch
        33 kB
        Matt Foley
      3. IBR_shortcut_v6atrunk.patch
        49 kB
        Matt Foley
      4. IBR_shortcut_v4atrunk.patch
        49 kB
        Matt Foley
      5. IBR_shortcut_v4atrunk.patch
        49 kB
        Matt Foley
      6. IBR_shortcut_v4atrunk.patch
        49 kB
        Matt Foley
      7. IBR_shortcut_v3atrunk.patch
        28 kB
        Matt Foley
      8. IBR_shortcut_v2a.patch
        29 kB
        Matt Foley
      9. HDFS-1295_for_ymerge.patch
        32 kB
        Matt Foley
      10. HDFS-1295_for_ymerge_v2.patch
        33 kB
        Matt Foley
      11. HDFS-1295_delta_for_trunk.patch
        2 kB
        Matt Foley

        Issue Links

          Activity

          No work has yet been logged on this issue.

            People

            • Assignee:
              Matt Foley
              Reporter:
              dhruba borthakur
            • Votes:
              0 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development