[HDFS-7916] 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infinite loop

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.7.0
• Fix Version/s: 2.8.0, 2.7.1, 3.0.0-alpha1
• Component/s: datanode
• Hadoop Flags: Reviewed
• Assignee: Rushabh Shah
• Reporter: Vinayakumar B
• Created: 11/Mar/15 10:40
• Updated: 06/Jan/17 00:44
• Resolved: 11/May/15 19:35

Description

if any badblock found, then BPSA for StandbyNode will go for infinite times to report it.

2015-03-11 19:43:41,528 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Failed to report bad block BP-1384821822-10.224.54.68-1422634566395:blk_1079544278_5812006 to namenode: stobdtserver3/10.224.54.70:18010
org.apache.hadoop.hdfs.server.datanode.BPServiceActorActionException: Failed to report bad block BP-1384821822-10.224.54.68-1422634566395:blk_1079544278_5812006 to namenode:
        at org.apache.hadoop.hdfs.server.datanode.ReportBadBlockAction.reportTo(ReportBadBlockAction.java:63)
        at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.processQueueMessages(BPServiceActor.java:1020)
        at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:762)
        at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:856)
        at java.lang.Thread.run(Thread.java:745)

Issue Links

• is broken by: HDFS-7704 DN heartbeat to Active NN may be blocked and expire if connection to Standby NN continues to time out.