[HDFS-15493] Update block map and name cache in parallel while loading fsimage.

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.1, 3.4.0
• Fix Version/s: 3.3.1, 3.4.0
• Component/s: namenode
• Target Version/s: 3.4.0
• Hadoop Flags: Reviewed
• Assignee: Chengwei Wang
• Reporter: Chengwei Wang
• Created: 24/Jul/20 08:49
• Updated: 27/Jan/24 03:15
• Resolved: 12/Aug/20 08:56

Description

While loading INodeDirectorySection of fsimage, it will update name cache and block map after added inode file to inode directory. It would reduce time cost of fsimage loading to enable these steps run in parallel.

In our test case, with patch HDFS-13694 and HDFS-14617, the time cost to load fsimage (220M files & 240M blocks) is 470s, with this patch , the time cost reduce to 410s.

Issue Links

• relates to: HDFS-13694 Making md5 computing being in parallel with image loading
• relates to: HDFS-14617 Improve fsimage load time by writing sub-sections to the fsimage index