[HDFS-7928] Scanning blocks from disk during rolling upgrade startup takes a lot of time if disks are busy

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.6.0
• Fix Version/s: 2.8.0, 3.0.0-alpha1
• Component/s: datanode
• Target Version/s: 2.8.0
• Hadoop Flags: Reviewed
• Assignee: Rushabh Shah
• Reporter: Rushabh Shah
• Created: 13/Mar/15 13:40
• Updated: 22/Dec/21 03:47
• Resolved: 25/Mar/15 20:04

Description

We observed this issue in rolling upgrade to 2.6.x on one of our cluster.
One of the disks was very busy and it took long time to scan that disk compared to other disks.
Seeing the sar (System Activity Reporter) data we saw that the particular disk was very busy performing IO operations.
Requesting for an improvement during datanode rolling upgrade.
During shutdown, we can persist the whole volume map on the disk and let the datanode read that file and create the volume map during startup after rolling upgrade.
This will not require the datanode process to scan all the disk and read the block.
This will significantly improve the datanode startup time.