[HIVE-23443] LLAP speculative task pre-emption seems to be not working

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.0.0-alpha-1
• Labels: pull-request-available
• Assignee: Prasanth Jayachandran Assign to me
• Reporter: Prasanth Jayachandran
• Created: 11/May/20 19:56
• Updated: 17/Nov/22 08:48
• Resolved: 17/May/20 16:02

Description

I think after HIVE-23210 we are getting a stable sort order and it is causing pre-emption to not work in certain cases.

"attempt_1589167813851_0000_119_01_000008_0 (hive_20200511055921_89598f09-19f1-4969-ab7a-82e2dd796273-119/Map 1, started at 2020-05-11 05:59:22, in preemption queue, can finish)", 
"attempt_1589167813851_0008_84_01_000008_1 (hive_20200511055928_7ae29ca3-e67d-4d1f-b193-05651023b503-84/Map 1, started at 2020-05-11 06:00:23, in preemption queue, can finish)" 

Scheduler only peek's at the pre-emption queue and looks at whether it is non-finishable. 

https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorService.java#L420

In the above case, all tasks are speculative but state change is not triggering pre-emption queue re-ordering so peek() always returns canFinish task even though non-finishable tasks are in the queue. 

Issue Links

• is related to: HIVE-23472 LLAP Guaranteed state update should trigger queue re-ordering
• relates to: HIVE-23210 Fix shortestjobcomparator when jobs submitted have 1 task their vertices
• relates to: HIVE-23238 FIX PreemptionQueueComparator edge cases
• links to: GitHub Pull Request #1012
• links to: GitHub Pull Request #1013
• links to: PR