[YARN-8815] RM fails to recover finished unmanaged AM

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 3.1.1
• Fix Version/s: 3.2.0, 3.1.2
• Target Version/s: 3.2.0, 3.1.2
• Hadoop Flags: Reviewed
• Assignee: Bibin Chundatt
• Reporter: Rakesh Shah
• Created: 24/Sep/18 11:02
• Updated: 25/Sep/18 11:11
• Resolved: 25/Sep/18 06:12

Description

 

while running a un managed am jar and restarting the RM - RM goes into standby

Below is the exception trace--

org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid resource request, no resources requested
 at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.validateAndCreateResourceRequest(RMAppManager.java:510)
 at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:389)
 at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:359)
 at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:589)
 at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1483)
 at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:848)
 at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
 at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:1224)
 at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1265)
 at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1261)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:422)
 at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
 at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1261)
 at org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:320)
 at org.apache.hadoop.yarn.server.resourcemanager.ActiveStandbyElectorBasedElectorService.becomeActive(ActiveStandbyElectorBasedElectorService.java:144)
 at org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:896)
 at org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:476)
 at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:728)
 at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:600)
2018-09-24 18:48:23,855 INFO org.apache.hadoop.service.AbstractService: Service RMActiveServices failed in state STARTED
org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid resource request, no resources requested
 at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.validateAndCreateResourceRequest(RMAppManager.java:510)
 at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:389)
 at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:359)
 at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:589)
 at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1483)
 at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:848)
 at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
 at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:1224)
 at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1265)
 at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1261)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:422)
 at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
 at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1261)
 at org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:320)
 at org.apache.hadoop.yarn.server.resourcemanager.ActiveStandbyElectorBasedElectorService.becomeActive(ActiveStandbyElectorBasedElectorService.java:144)
 at org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:896)
 at org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:476)
 at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:728)
 at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:600)

Issue Links

• is broken by: YARN-5028 RMStateStore should trim down app state for completed applications