[HDFS-4654] FileNotFoundException: ID mismatch

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 3.0.0-alpha1
• Component/s: ha, namenode
• Assignee: Brandon Li
• Reporter: Fengdong Yu
• Created: 01/Apr/13 03:35
• Updated: 12/May/16 18:19
• Resolved: 11/Apr/13 02:57

Description

Mu cluster was build from source code trunk r1463074.

I got an exception as follows when I put a file to the HDFS.

13/04/01 09:33:45 WARN retry.RetryInvocationHandler: Exception while invoking addBlock of class ClientNamenodeProtocolTranslatorPB. Trying to fail over immediately.
13/04/01 09:33:45 WARN hdfs.DFSClient: DataStreamer Exception
java.io.FileNotFoundException: ID mismatch. Request id and saved id: 1073 , 1050
at org.apache.hadoop.hdfs.server.namenode.INodeId.checkId(INodeId.java:51)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2501)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.analyzeFileState(FSNamesystem.java:2298)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2212)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:498)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:356)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:40979)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:526)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1018)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1818)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1814)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1489)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1812)

please reproduce as :

hdfs dfs -put test.data /user/data/test.data
after this command start to run, then kill active name node process.

I have only three nodes(A,B,C) for test
A and B are name nodes.
B and C are data nodes.
ZK deployed on A, B and C.

A, B and C are all journal nodes.

Thanks.

Issue Links

• relates to: HDFS-4339 Persist inode id in fsimage and editlog