Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-7784

load fsimage in parallel

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: namenode
    • Labels:

      Description

      When single Namenode has huge amount of files, without using federation, the startup/restart speed is slow. The fsimage loading step takes the most of the time. fsimage loading can seperate to two parts, deserialization and object construction(mostly map insertion). Deserialization takes the most of CPU time. So we can do deserialization in parallel, and add to hashmap in serial. It will significantly reduce the NN start time.

        Attachments

        1. HDFS-7784.001.patch
          23 kB
          Walter Su
        2. test-20150213.pdf
          127 kB
          Walter Su

          Issue Links

            Activity

              People

              • Assignee:
                walter.k.su Walter Su
                Reporter:
                walter.k.su Walter Su
              • Votes:
                0 Vote for this issue
                Watchers:
                32 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: