[HIVE-22113] Prevent LLAP shutdown on AMReporter related RuntimeException

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.1
• Component/s: llap
• Labels: llap
• Assignee: Oliver Draese
• Reporter: Oliver Draese
• Created: 14/Aug/19 18:34
• Updated: 01/Dec/22 09:02
• Resolved: 16/Aug/19 01:40

Description

If a task attempt cannot be removed from AMReporter (i.e. task attempt was not found), the AMReporter throws a RuntimeException. This exception is not caught and trickles up, causing an LLAP shutdown:
2019-08-08T23:34:39,748&nbspERROR&nbsp[Wait-Queue-Scheduler-0&nbsp()]&nbsporg.apache.hadoop.hive.llap.daemon.impl.LlapDaemon:&nbspThread&nbspThread[Wait-Queue-Scheduler-0,5,main]&nbspthrew&nbspan&nbspException.&nbspShutting&nbspdown&nbspnow...java.lang.RuntimeException:&nbspattempt_1563528877295_18872_3728_01_000003_0&nbspwas&nbspnot&nbspregistered&nbspand&nbspcouldn't&nbspbe&nbspremoved    at&nbsporg.apache.hadoop.hive.llap.daemon.impl.AMReporter$AMNodeInfo.removeTaskAttempt(AMReporter.java:524)&nbsp~[hive-llap-server-3.1.0.3.1.0.103-1.jar:3.1.0.3.1.0.103-1]    at&nbsporg.apache.hadoop.hive.llap.daemon.impl.AMReporter.unregisterTask(AMReporter.java:243)&nbsp~[hive-llap-server-3.1.0.3.1.0.103-1.jar:3.1.0.3.1.0.103-1]    at&nbsporg.apache.hadoop.hive.llap.daemon.impl.TaskRunnerCallable.killTask(TaskRunnerCallable.java:384)&nbsp~[hive-llap-server-3.1.0.3.1.0.103-1.jar:3.1.0.3.1.0.103-1]    at&nbsporg.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService.handleScheduleAttemptedRejection(TaskExecutorService.java:739)&nbsp~[hive-llap-server-3.1.0.3.1.0.103-1.jar:3.1.0.3.1.0.103-1]    at&nbsporg.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService.access$1100(TaskExecutorService.java:91)&nbsp~[hive-llap-server-3.1.0.3.1.0.103-1.jar:3.1.0.3.1.0.103-1]    at&nbsporg.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService$WaitQueueWorker.run(TaskExecutorService.java:396)&nbsp~[hive-llap-server-3.1.0.3.1.0.103-1.jar:3.1.0.3.1.0.103-1]    at&nbspjava.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)&nbsp~[?:1.8.0_161]    at&nbspcom.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)&nbsp[hive-exec-3.1.0.3.1.0.103-1.jar:3.1.0-SNAPSHOT]    at&nbspcom.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)&nbsp[hive-exec-3.1.0.3.1.0.103-1.jar:3.1.0-SNAPSHOT]    at&nbspcom.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)&nbsp[hive-exec-3.1.0.3.1.0.103-1.jar:3.1.0-SNAPSHOT]    at&nbspjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)&nbsp[?:1.8.0_161]    at&nbspjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)&nbsp[?:1.8.0_161]    at&nbspjava.lang.Thread.run(Thread.java:748)&nbsp[?:1.8.0_161]

Issue Links

• Is contained by: HIVE-26751 Bug Fixes and Improvements for 3.2.0 release