[HDFS-7784] load fsimage in parallel

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Duplicate
• Component/s: namenode
• Labels: BB2015-05-TBR
• Assignee: Walter Su
• Reporter: Walter Su
• Created: 12/Feb/15 06:22
• Updated: 18/Nov/19 23:19
• Resolved: 18/Nov/19 23:19

Description

When single Namenode has huge amount of files, without using federation, the startup/restart speed is slow. The fsimage loading step takes the most of the time. fsimage loading can seperate to two parts, deserialization and object construction(mostly map insertion). Deserialization takes the most of CPU time. So we can do deserialization in parallel, and add to hashmap in serial. It will significantly reduce the NN start time.

Issue Links

• is related to: HDFS-14617 Improve fsimage load time by writing sub-sections to the fsimage index
• is superceded by: HDFS-14617 Improve fsimage load time by writing sub-sections to the fsimage index