[HIVE-15731] sessions are not returned to the sessionPool in case of an interrupt

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 2.2.0
• Target Version/s: 2.2.0
• Assignee: Siddharth Seth
• Reporter: Siddharth Seth
• Created: 25/Jan/17 23:34
• Updated: 25/Jul/17 20:04
• Resolved: 26/Jan/17 04:54

Description

While returning a session to the pool, the interrupt status on the thread seems to be set, which causes the pool return to fail.
The session slot is useless at this point. A HS2 instance configured for a single session will stop running queries.

2017-01-25T01:21:02,803 ERROR [HiveServer2-Background-Pool: Thread-117]: exec.Task (TezTask.java:execute(210)) - Failed to execute tez graph.
java.lang.InterruptedException
  at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220) ~[?:1.8.0_77]
  at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335) ~[?:1.8.0_77]
  at java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:350) ~[?:1.8.0_77]
  at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.returnSession(TezSessionPoolManager.java:337) ~[hive-exec-2.1.jar:2.1]
  at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:196) [hive-exec-2.1.jar:2.1]
  at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197) [hive-exec-2.1.jar:2.1]
  at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100) [hive-exec-2.1.jar:2.1]
  at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1874) [hive-exec-2.1.jar:2.1]
  at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1578) [hive-exec-2.1.jar:2.1]
  at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1329) [hive-exec-2.1.jar:2.1]
  at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1098) [hive-exec-2.1.jar:2.1]
  at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1091) [hive-exec-2.1.jar:2.1]
  at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:242) [hive-service-2.1.jar:2.1]
  at org.apache.hive.service.cli.operation.SQLOperation.access$800(SQLOperation.java:91) [hive-service-2.1.jar:2.1]
  at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:334) [hive-service-2.1.jar:2.1]
  at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_77]
  at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_77]
  at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724) [hadoop-common-2.7.3.2.6.0.0-389.jar:?]
  at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:347) [hive-service-2.1.jar:2.1]
  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_77]
  at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_77]
  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_77]
  at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_77]
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_77]
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_77]
  at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
2017-01-25T01:21:02,803 INFO  [e208c68c-f502-4c9d-a7b1-f3e7dd8b9447 HiveServer2-Handler-Pool: Thread-99]: session.SessionState (SessionState.java:resetThreadName(428))