[HDFS-854] Datanode should scan devices in parallel to generate block report

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.22.0
• Fix Version/s: 0.21.0
• Component/s: datanode
• Hadoop Flags: Reviewed
• Assignee: Dmytro Molkov
• Reporter: Dhruba Borthakur
• Created: 27/Dec/09 07:14
• Updated: 07/Oct/10 01:15
• Resolved: 30/Mar/10 03:16

Description

A Datanode should scan its disk devices in parallel so that the time to generate a block report is reduced. This will reduce the startup time of a cluster.
A datanode has 12 disk (each of 1 TB) to store HDFS blocks. There is a total of 150K blocks on these 12 disks. It takes the datanode upto 20 minutes to scan these devices to generate the first block report.

Issue Links

• is related to: HDFS-270 DFS Upgrade should process dfs.data.dirs in parallel