[HIVE-15693] LLAP: cached threadpool in AMReporter creates too many threads leading to OOM

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 2.2.0
• Component/s: llap
• Labels: TODOC2.2
• Assignee: Rajesh Balamohan
• Reporter: Rajesh Balamohan
• Created: 23/Jan/17 03:59
• Updated: 25/Jul/17 20:04
• Resolved: 27/Jan/17 23:12

Description

branch: master

2017-01-22T19:52:42,470 WARN  [IPC Server handler 3 on 34642 ()] org.apache.hadoop.ipc.Server: IPC Server handler 3 on 34642, call org.apache.hadoop.hive.llap.protocol.LlapProtocolBlockingPB.submitWork ...Call#17257 Retry#0
java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method) ~[?:1.8.0_77]
        at java.lang.Thread.start(Thread.java:714) [?:1.8.0_77]
        at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950) ~[?:1.8.0_77]
        at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1368) ~[?:1.8.0_77]
        at com.google.common.util.concurrent.MoreExecutors$ListeningDecorator.execute(MoreExecutors.java:480) ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
        at com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:61) ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
        at org.apache.hadoop.hive.llap.daemon.impl.AMReporter.taskKilled(AMReporter.java:231) ~[hive-llap-server-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
        at org.apache.hadoop.hive.llap.daemon.impl.ContainerRunnerImpl$KilledTaskHandlerImpl.taskKilled(ContainerRunnerImpl.java:501) ~[hive-llap-server-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]

Issue Links

• relates to: HIVE-15733 LLAP AMHeartbeat threads should be restricted per app