[HDFS-3874] Exception when client reports bad checksum to NN

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.0.0-alpha, 0.23.5
• Component/s: hdfs-client, namenode
• Target Version/s: 2.0.3-alpha, 0.23.7
• Assignee: Kihwal Lee
• Reporter: Todd Lipcon
• Created: 30/Aug/12 22:10
• Updated: 28/Jan/13 18:41
• Resolved: 28/Jan/13 18:41

Description

We see the following exception in our logs on a cluster:

2012-08-27 16:34:30,400 INFO org.apache.hadoop.hdfs.StateChange: *DIR* NameNode.reportBadBlocks
2012-08-27 16:34:30,400 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:hdfs (auth:SIMPLE) cause:java.io.IOException: Cannot mark blk_8285012733733669474_140475196{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[172.29.97.219:50010|RBW]]}(same as stored) as corrupt because datanode :0 does not exist
2012-08-27 16:34:30,400 INFO org.apache.hadoop.ipc.Server: IPC Server handler 46 on 8020, call org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol.reportBadBlocks from 172.29.97.219:43805: error: java.io.IOException: Cannot mark blk_8285012733733669474_140475196{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[172.29.97.219:50010|RBW]]}(same as stored) as corrupt because datanode :0 does not exist
java.io.IOException: Cannot mark blk_8285012733733669474_140475196{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[172.29.97.219:50010|RBW]]}(same as stored) as corrupt because datanode :0 does not exist
        at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.markBlockAsCorrupt(BlockManager.java:1001)
        at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.findAndMarkBlockAsCorrupt(BlockManager.java:994)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.reportBadBlocks(FSNamesystem.java:4736)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.reportBadBlocks(NameNodeRpcServer.java:537)
        at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolServerSideTranslatorPB.reportBadBlocks(DatanodeProtocolServerSideTranslatorPB.java:242)
        at org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos$DatanodeProtocolService$2.callBlockingMethod(DatanodeProtocolProtos.java:20032)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)

Issue Links

• is duplicated by: HDFS-3875 Issue handling checksum errors in write pipeline