[HDFS-270] DFS Upgrade should process dfs.data.dirs in parallel

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 0.20.2
• Component/s: datanode
• Tags: datanode startup, volume parallel
• Assignee: Hairong Kuang
• Reporter: Stu Hood
• Created: 10/Nov/07 04:00
• Updated: 17/Jul/14 18:49
• Resolved: 17/Jul/14 18:49

Description

I just upgraded from 0.14.2 to 0.15.0, and things went very smoothly, if a little slowly.

The main reason the upgrade took so long was the block upgrades on the datanodes. Each of our datanodes has 3 drives listed for the dfs.data.dir parameter. From looking at the logs, it is fairly clear that the upgrade procedure does not attempt to upgrade all listed dfs.data.dir's in parallel.

I think even if all of your dfs.data.dir's are on the same physical device, there would still be an advantage to performing the upgrade process in parallel. The less downtime, the better: especially if it is potentially 20 minutes versus 60 minutes.

Issue Links

• relates to: HDFS-854 Datanode should scan devices in parallel to generate block report