[YARN-8001] Newly created Yarn application ID lost after RM failover

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.7.3, 2.9.0
• Component/s: RM
• Assignee: Unassigned
• Reporter: shanyu zhao
• Created: 05/Mar/18 23:50
• Updated: 23/Feb/22 10:26

Description

I’ve seen a problem in Hadoop 2.7.3 where the newly submitted yarn application was lost after a RM failover. It looks like when handling Application submission, RM does not write it to the state-store (We are using zookeeper based state store) immediately before it respond to the client. But later it failed over to another RM and all write call to the state store failed. The new RM recovers state from the state-store, and this app is lost. 

 

The symptom is error message at client side claiming a previously submitted application ID does not exist:

2018-02-22 14:54:50,258 [JobControl] WARN  org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider - Invocation returned exception on [rm1] : org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException: Application with id 'application_1519310222933_0160' doesn't exist in RM. Please check that the job submission was successful.

 

This is a timeline excerpted from the resource manager logs:

2018-02-22 14:54:06.7685260    headnode1        Storing application with id application_1519310222933_0160

2018-02-22 14:54:06.7685660    headnode1              application_1519310222933_0160 State change from NEW to NEW_SAVING

2018-02-22 14:54:17.8924760    headnode1        Transitioning to standby state

2018-02-22 14:54:30.3951160    headnode0        Transitioning to active state