[HIVE-10692] LLAP: DAGs get stuck at start with no tasks executing

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Cannot Reproduce
• Assignee: Siddharth Seth Assign to me
• Reporter: Sergey Shelukhin
• Created: 12/May/15 21:11
• Updated: 04/Aug/15 01:08
• Resolved: 04/Aug/15 01:08

Description

Internal app ID application_1429683757595_0914, LLAP application_1429683757595_0913. If someone without access wants to investigate I'll get the logs.
2nd dag failed to start executing:
See syslog_dag_1429683757595_0914_2 log file.
This happened to me a couple of times today, didn't see it before.

After many S_TA_LAUNCH_REQUEST-s, the following is logged and after that there's no more logging aside from refreshes until I killed the DAG. LLAP daemons were idling meanwhile.
I don't see any errors (aside from ATS) before this happened

2015-05-12 13:52:08,997 INFO [TaskSchedulerEventHandlerThread] rm.TaskSchedulerEventHandler: Processing the event EventType: S_TA_LAUNCH_REQUEST
2015-05-12 13:52:18,507 INFO [LlapSchedulerNodeEnabler] impl.LlapYarnRegistryImpl: Starting to refresh ServiceInstanceSet 556007888
2015-05-12 13:52:25,315 INFO [HistoryEventHandlingThread] ats.ATSHistoryLoggingService: Event queue stats, eventsProcessedSinceLastUpdate=407, eventQueueSize=614
2015-05-12 13:52:28,507 INFO [LlapSchedulerNodeEnabler] impl.LlapYarnRegistryImpl: Starting to refresh ServiceInstanceSet 556007888