Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-22113

Prevent LLAP shutdown on AMReporter related RuntimeException

Log workAgile BoardRank to TopRank to BottomBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.1.1
    • None
    • llap

    Description

      If a task attempt cannot be removed from AMReporter (i.e. task attempt was not found), the AMReporter throws a RuntimeException. This exception is not caught and trickles up, causing an LLAP shutdown:
      2019-08-08T23:34:39,748&nbspERROR&nbsp[Wait-Queue-Scheduler-0&nbsp()]&nbsporg.apache.hadoop.hive.llap.daemon.impl.LlapDaemon:&nbspThread&nbspThread[Wait-Queue-Scheduler-0,5,main]&nbspthrew&nbspan&nbspException.&nbspShutting&nbspdown&nbspnow...java.lang.RuntimeException:&nbspattempt_1563528877295_18872_3728_01_000003_0&nbspwas&nbspnot&nbspregistered&nbspand&nbspcouldn't&nbspbe&nbspremoved    at&nbsporg.apache.hadoop.hive.llap.daemon.impl.AMReporter$AMNodeInfo.removeTaskAttempt(AMReporter.java:524)&nbsp~[hive-llap-server-3.1.0.3.1.0.103-1.jar:3.1.0.3.1.0.103-1]    at&nbsporg.apache.hadoop.hive.llap.daemon.impl.AMReporter.unregisterTask(AMReporter.java:243)&nbsp~[hive-llap-server-3.1.0.3.1.0.103-1.jar:3.1.0.3.1.0.103-1]    at&nbsporg.apache.hadoop.hive.llap.daemon.impl.TaskRunnerCallable.killTask(TaskRunnerCallable.java:384)&nbsp~[hive-llap-server-3.1.0.3.1.0.103-1.jar:3.1.0.3.1.0.103-1]    at&nbsporg.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService.handleScheduleAttemptedRejection(TaskExecutorService.java:739)&nbsp~[hive-llap-server-3.1.0.3.1.0.103-1.jar:3.1.0.3.1.0.103-1]    at&nbsporg.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService.access$1100(TaskExecutorService.java:91)&nbsp~[hive-llap-server-3.1.0.3.1.0.103-1.jar:3.1.0.3.1.0.103-1]    at&nbsporg.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService$WaitQueueWorker.run(TaskExecutorService.java:396)&nbsp~[hive-llap-server-3.1.0.3.1.0.103-1.jar:3.1.0.3.1.0.103-1]    at&nbspjava.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)&nbsp~[?:1.8.0_161]    at&nbspcom.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)&nbsp[hive-exec-3.1.0.3.1.0.103-1.jar:3.1.0-SNAPSHOT]    at&nbspcom.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)&nbsp[hive-exec-3.1.0.3.1.0.103-1.jar:3.1.0-SNAPSHOT]    at&nbspcom.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)&nbsp[hive-exec-3.1.0.3.1.0.103-1.jar:3.1.0-SNAPSHOT]    at&nbspjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)&nbsp[?:1.8.0_161]    at&nbspjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)&nbsp[?:1.8.0_161]    at&nbspjava.lang.Thread.run(Thread.java:748)&nbsp[?:1.8.0_161]

      Attachments

        1. HIVE-22113.1.patch
          1 kB
          Oliver Draese
        2. HIVE-22113.2.patch
          1 kB
          Oliver Draese
        3. HIVE-22113.patch
          1 kB
          Oliver Draese

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            odraese Oliver Draese Assign to me
            odraese Oliver Draese
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment