[HDFS-7430] Rewrite the BlockScanner to use O(1) memory and use multiple threads

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.7.0
• Fix Version/s: 2.7.0
• Target Version/s: 2.7.0
• Assignee: Colin McCabe
• Reporter: Colin McCabe
• Created: 24/Nov/14 04:35
• Updated: 28/Jul/17 07:45
• Resolved: 22/Jan/15 03:03

Description

We should update the BlockScanner to use a constant amount of memory by keeping track of what block was scanned last, rather than by tracking the scan status of all blocks in memory. Also, instead of having just one thread, we should have a verification thread per hard disk (or other volume), scanning at a configurable rate of bytes per second.

Issue Links

• breaks: HDFS-9190 VolumeScanner throwing NPE while scanning suspect block.
• breaks: HDFS-7633 BlockPoolSliceScanner fails when Datanode has too many blocks
• breaks: HDFS-7686 Re-add rapid rescan of possibly corrupt block feature to the block scanner
• breaks: HDFS-8681 BlockScanner is incorrectly disabled by default
• breaks: HDFS-12209 VolumeScanner scan cursor not save periodic
• incorporates: HDFS-6111 Avoid duplicate entries in Block Scan verification logs
• incorporates: HDFS-6147 New blocks scanning will be delayed due to issue in BlockPoolSliceScanner#updateBytesToScan(..)
• is duplicated by: HDFS-7532 dncp_block_verification.log.prev too large
• is duplicated by: HDFS-8028 TestNNHandlesBlockReportPerStorage/TestNNHandlesCombinedBlockReport Failed after patched HDFS-7704
• is related to: HDFS-8691 Cleanup BlockScanner initialization and add test for configuration contract
• is related to: HDFS-3488 BlockPoolSliceScanner#getNewBlockScanTime does not handle numbers > 31 bits properly
• is related to: HDFS-7721 The HDFS BlockScanner may run fast during the first hour
• supercedes: HDFS-4224 The dncp_block_verification log can be compressed