[HIVE-15529] LLAP: TaskSchedulerService can get stuck when scheduling tasks as disabled node is not re-enabled in NodeEnablerCallable

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 2.2.0
• Component/s: llap
• Hadoop Flags: Reviewed
• Assignee: Rajesh Balamohan
• Reporter: Rajesh Balamohan
• Created: 02/Jan/17 10:09
• Updated: 25/Jul/17 20:05
• Resolved: 04/Jan/17 00:55

Description

Easier way to simulate the issue:
1. Start hive cli with "--hiveconf hive.execution.mode=llap"
2. Run a sql script file (e.g sql script containing tpc-ds queries)
3. In the middle of the run, press "ctrl+C" which would interrupt the current job. This should not exit the hive cli yet.
4. After sometime, launch the same SQL script in same cli. This would get stuck indefinitely (waiting for computing the splits).

Even when cli is quit, AM runs forever until explicitly killed.

Issue seems to be around LlapTaskSchedulerService::schedulePendingTasks dealing with the loop when it encounters DELAYED_RESOURCES on task scheduling.