[MAPREDUCE-6363] [NNBench] Lease mismatch error when running with multiple mappers

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 2.8.0, 2.7.3, 2.6.4, 3.0.0-alpha1
• Component/s: benchmarks
• Target Version/s: 2.8.0, 2.6.4
• Hadoop Flags: Reviewed
• Assignee: Bibin Chundatt
• Reporter: Brahma Reddy Battula
• Created: 24/Jun/13 13:15
• Updated: 08/Nov/17 04:40
• Resolved: 22/Mar/16 16:46

Description

Command :

./yarn jar ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.0.1-tests.jar nnbench -operation create_write -numberOfFiles 1000 -blockSize 268435456 -bytesToWrite 1024000000 -baseDir /benchmarks/NNBench`hostname -s` -replicationFactorPerFile 3 -maps 100 -reduces 10

Trace :

013-06-21 10:44:53,763 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 9005, call org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from 192.168.105.214:36320: error: org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: Lease mismatch on /benchmarks/NNBenchlinux-185/data/file_linux-214__0 owned by DFSClient_attempt_1371782327901_0001_m_000048_0_1383437860_1 but is accessed by DFSClient_attempt_1371782327901_0001_m_000084_0_1880545303_1
org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: Lease mismatch on /benchmarks/NNBenchlinux-185/data/file_linux-214__0 owned by DFSClient_attempt_1371782327901_0001_m_000048_0_1383437860_1 but is accessed by DFSClient_attempt_1371782327901_0001_m_000084_0_1880545303_1
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2351)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.analyzeFileState(FSNamesystem.java:2098)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2019)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:501)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:213)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:52012)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:435)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:925)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1710)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1706)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)

Issue Links

• breaks: MAPREDUCE-6656 [NNBench] OP_DELETE operation isn't working after MAPREDUCE-6363
• contains: MAPREDUCE-6465 NNBench result wrong when number of reducers greater than 1
• duplicates: HADOOP-15020 NNBench not support run more than one map task on the same host