[TEZ-3468] Dispatcher error on appmaster shutdown: Cannot get ApplicationACLs before all services have started

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Unassigned
• Reporter: Kavan Suresh
• Created: 13/Oct/16 21:47
• Updated: 13/Oct/16 22:28

Description

Tez AM throws a Dispatcher error during shutdown when running LLAP queries.

2016-10-13 09:14:24,999 [INFO] [LlapScheduler] |tezplugins.LlapTaskSchedulerService|: ScheduleResult for Task: TaskInfo{task=attempt_1476340082683_0013_137_01_000019_0, priority=14, startTime=0, containerId=null, assignedInstance=null, uniqueId=7505, localityDelayTimeout=9223372036854775807} = DELAYED_RESOURCES
2016-10-13 09:14:24,999 [INFO] [LlapScheduler] |tezplugins.LlapTaskSchedulerService|: Preempting for task=attempt_1476340082683_0013_137_01_000019_0 on any available host
2016-10-13 09:14:24,999 [INFO] [LlapScheduler] |tezplugins.LlapTaskSchedulerService|: No tasks qualify as killable to schedule tasks at priority 14
2016-10-13 09:14:24,999 [INFO] [Socket Reader #1 for port 36800] |ipc.Server|: Auth successful for application_1476340082683_0013 (auth:SIMPLE)
2016-10-13 09:14:24,999 [ERROR] [Dispatcher thread {Central}] |common.AsyncDispatcher|: Error in dispatcher thread
org.apache.tez.dag.api.TezUncheckedException: Cannot get ApplicationACLs before all services have started
        at org.apache.tez.dag.app.DAGAppMaster$RunningAppContext.getApplicationACLs(DAGAppMaster.java:1711)
        at org.apache.tez.dag.app.rm.container.AMContainerImpl$LaunchRequestTransition.transition(AMContainerImpl.java:491)
        at org.apache.tez.dag.app.rm.container.AMContainerImpl$LaunchRequestTransition.transition(AMContainerImpl.java:448)
        at org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:362)
        at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
        at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
        at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
        at org.apache.tez.dag.app.rm.container.AMContainerImpl.handle(AMContainerImpl.java:419)
        at org.apache.tez.dag.app.rm.container.AMContainerImpl.handle(AMContainerImpl.java:76)
        at org.apache.tez.dag.app.rm.container.AMContainerMap.handle(AMContainerMap.java:60)
        at org.apache.tez.dag.app.rm.container.AMContainerMap.handle(AMContainerMap.java:36)
        at org.apache.tez.common.AsyncDispatcher.dispatch(AsyncDispatcher.java:183)
        at org.apache.tez.common.AsyncDispatcher$1.run(AsyncDispatcher.java:114)
        at java.lang.Thread.run(Thread.java:745)
2016-10-13 09:14:24,999 [WARN] [Dispatcher thread {Central}] |container.AMContainerImpl|: Unexpected TA Assignment: TAId: attempt_1476340082683_0013_137_01_000017_3  for ContainerId: container_222212222_0013_01_007508 while in state: ALLOCATED