[HDFS-11086] DataNode disk check improvements

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Done
• Component/s: datanode
• Assignee: Arpit Agarwal
• Reporter: Arpit Agarwal
• Created: 01/Nov/16 21:51
• Updated: 06/Jun/17 13:38
• Resolved: 06/Jun/17 13:38

Description

This Jira tracks a few improvements to DataNode’s usage of DiskChecker to address the following problems:

1. Checks are serialized so a single slow disk can indefinitely delay checking the rest.
2. Related to 1, no detection of stalled checks.
3. Lack of granularity. A single IO error initiates checking all disks.
4. Inconsistent activation. Some DataNode IO failures trigger disk checks but not all.

Sub-Tasks

• Support for running async disk checks in DataNode: Resolved
• Support for parallel checking of StorageLocations on DataNode startup: Resolved
• Update DataNode to use StorageLocationChecker at startup: Resolved
• Support for parallel checking of FsVolumes: Resolved
• Update DataNode to use DatasetVolumeChecker: Resolved
• Implement equals and hashcode in FsVolumeSpi implementations: Resolved