[YARN-1058] Recovery issues on RM Restart with FileSystemRMStateStore

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.1.0-beta
• Component/s: resourcemanager
• Assignee: Karthik Kambatla
• Reporter: Karthik Kambatla
• Created: 12/Aug/13 08:25
• Updated: 03/Nov/14 18:33
• Resolved: 10/Oct/13 23:52

Description

App recovery doesn't work as expected using FileSystemRMStateStore.

Steps to reproduce:

• Ran sleep job with a single map and sleep time of 2 mins
• Restarted RM while the map task is still running
• The first attempt fails with the following error

  Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): Password not found for ApplicationAttempt appattempt_1376294441253_0001_000001
  	at org.apache.hadoop.ipc.Client.call(Client.java:1404)
  	at org.apache.hadoop.ipc.Client.call(Client.java:1357)
  	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
  	at $Proxy28.finishApplicationMaster(Unknown Source)
  	at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.finishApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:91)
  

• The second attempt fails with a different error:

  Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): No lease on /tmp/hadoop-yarn/staging/kasha/.staging/job_1376294441253_0001/job_1376294441253_0001_2.jhist: File does not exist. Holder DFSClient_NONMAPREDUCE_389533538_1 does not have any open files.
  	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2737)
  	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.analyzeFileState(FSNamesystem.java:2543)
  	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2454)
  	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:534)
  	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:387)
  	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:48073)
  	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:605)
  

Issue Links

• depends upon: YARN-1116 Populate AMRMTokens back to AMRMTokenSecretManager after RM restarts