[HDFS-13694] Making md5 computing being in parallel with image loading

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.10.0, 3.0.4, 3.3.0, 3.2.1, 2.9.3, 3.1.3
• Hadoop Flags: Reviewed
• Assignee: Lisheng Sun
• Reporter: zhouyingchao
• Created: 22/Jun/18 08:45
• Updated: 24/Jul/20 09:37
• Resolved: 05/Jul/19 17:24

Description

During namenode image loading, it firstly compute the md5 and then load the image. Actually these two steps can be in parallel.

 Test this patch against a fsimage of a 70PB 2.4 cluster (200million files and 300million blocks), the image loading time be reduced from 1210 seconds to 1105 seconds.So it can reduce up to about 10% of time.

 

Issue Links

• is related to: HDFS-15493 Update block map and name cache in parallel while loading fsimage.
• links to: GitHub Pull Request #1010