[HDFS-10627] Volume Scanner marks a block as "suspect" even if the exception is network-related

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.7.0
• Fix Version/s: 2.7.4, 3.0.0-alpha2
• Component/s: hdfs
• Target Version/s: 2.7.4
• Hadoop Flags: Reviewed
• Assignee: Rushabh Shah
• Reporter: Rushabh Shah
• Created: 14/Jul/16 16:35
• Updated: 20/Sep/17 16:47
• Resolved: 20/Oct/16 16:11

Description

In the BlockSender code,

BlockSender.java

        if (!ioem.startsWith("Broken pipe") && !ioem.startsWith("Connection reset")) {
          LOG.error("BlockSender.sendChunks() exception: ", e);
        }
        datanode.getBlockScanner().markSuspectBlock(
              volumeRef.getVolume().getStorageID(),
              block);

Before HDFS-7686, the block was marked as suspect only if the exception message doesn't start with Broken pipe or Connection reset.
But after HDFS-7686, the block is marked as corrupt irrespective of the exception message.

In one of our datanode, it took approximately a whole day (22 hours) to go through all the suspect blocks to scan one corrupt block.

Issue Links

• is broken by: HDFS-7686 Re-add rapid rescan of possibly corrupt block feature to the block scanner
• relates to: HDFS-12514 Cancelled HedgedReads cause block to be marked as suspect on Windows