[HDFS-3931] TestDatanodeBlockScanner#testBlockCorruptionPolicy2 is broken

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.0.0-alpha
• Fix Version/s: 2.0.3-alpha
• Component/s: test
• Hadoop Flags: Reviewed
• Assignee: Andy Isaacson
• Reporter: Eli Collins
• Created: 12/Sep/12 18:38
• Updated: 15/Feb/13 13:11
• Resolved: 27/Sep/12 04:11

Description

Per Andy's comment on HDFS-3902:

TestDatanodeBlockScanner still fails about 1/5 runs in testBlockCorruptionRecoveryPolicy2. That's due to a separate test issue also uncovered by HDFS-3828.
The failure scenario for this one is a bit more tricky. I think I've captured the scenario below:

• The test corrupts 2/3 replicas.
• client reports a bad block.
• NN asks a DN to re-replicate, and randomly picks the other corrupt replica.
• DN notices the incoming replica is corrupt and reports it as a bad block, but does not inform the NN that re-replication failed.
• NN keeps the block on pendingReplications.
• BP scanner wakes up on both DNs with corrupt blocks, both report corruption. NN reports both as duplicates, one from the client and one from the DN report above.
  since block is on pendingReplications, NN does not schedule another replication.

Issue Links

• relates to: HDFS-3660 TestDatanodeBlockScanner#testBlockCorruptionRecoveryPolicy2 times out
• relates to: HDFS-3902 TestDatanodeBlockScanner#testBlockCorruptionPolicy is broken