[HIVE-13960] Session timeout may happen before HIVE_SERVER2_IDLE_SESSION_TIMEOUT for back-to-back synchronous operations.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.3.0
• Component/s: HiveServer2
• Hadoop Flags: Reviewed
• Assignee: Zhihai Xu
• Reporter: Zhihai Xu
• Created: 07/Jun/16 03:45
• Updated: 21/Jul/17 17:45
• Resolved: 11/Jun/16 16:15

Description

Session timeout may happen before HIVE_SERVER2_IDLE_SESSION_TIMEOUT(hive.server2.idle.session.timeout) for back-to-back synchronous operations.
This issue can happen with the following two operations op1 and op2: op2 is a synchronous long running operation, op2 is running right after op1 is closed.

1. closeOperation(op1) is called:
this will set lastIdleTime with value System.currentTimeMillis() because opHandleSet becomes empty after closeOperation remove op1 from opHandleSet.

2. op2 is running for long time by calling executeStatement right after closeOperation(op1) is called.
If op2 is running for more than HIVE_SERVER2_IDLE_SESSION_TIMEOUT, then the session will timeout even when op2 is still running.

We hit this issue when we use PyHive to execute non-async operation
The following is the exception we see:

File "/usr/local/lib/python2.7/dist-packages/pyhive/hive.py", line 126, in close
    _check_status(response)
  File "/usr/local/lib/python2.7/dist-packages/pyhive/hive.py", line 362, in _check_status
    raise OperationalError(response)
OperationalError: TCloseSessionResp(status=TStatus(errorCode=0, errorMessage='Session does not exist!', sqlState=None, infoMessages=['*org.apache.hive.service.cli.HiveSQLException:Session does not exist!:12:11', 'org.apache.hive.service.cli.session.SessionManager:closeSession:SessionManager.java:311', 'org.apache.hive.service.cli.CLIService:closeSession:CLIService.java:221', 'org.apache.hive.service.cli.thrift.ThriftCLIService:CloseSession:ThriftCLIService.java:471', 'org.apache.hive.service.cli.thrift.TCLIService$Processor$CloseSession:getResult:TCLIService.java:1273', 'org.apache.hive.service.cli.thrift.TCLIService$Processor$CloseSession:getResult:TCLIService.java:1258', 'org.apache.thrift.ProcessFunction:process:ProcessFunction.java:39', 'org.apache.thrift.TBaseProcessor:process:TBaseProcessor.java:39', 'org.apache.hive.service.auth.TSetIpAddressProcessor:process:TSetIpAddressProcessor.java:56', 'org.apache.thrift.server.TThreadPoolServer$WorkerProcess:run:TThreadPoolServer.java:285', 'java.util.concurrent.ThreadPoolExecutor:runWorker:ThreadPoolExecutor.java:1145', 'java.util.concurrent.ThreadPoolExecutor$Worker:run:ThreadPoolExecutor.java:615', 'java.lang.Thread:run:Thread.java:745'], statusCode=3))
TCloseSessionResp(status=TStatus(errorCode=0, errorMessage='Session does not exist!', sqlState=None, infoMessages=['*org.apache.hive.service.cli.HiveSQLException:Session does not exist!:12:11', 'org.apache.hive.service.cli.session.SessionManager:closeSession:SessionManager.java:311', 'org.apache.hive.service.cli.CLIService:closeSession:CLIService.java:221', 'org.apache.hive.service.cli.thrift.ThriftCLIService:CloseSession:ThriftCLIService.java:471', 'org.apache.hive.service.cli.thrift.TCLIService$Processor$CloseSession:getResult:TCLIService.java:1273', 'org.apache.hive.service.cli.thrift.TCLIService$Processor$CloseSession:getResult:TCLIService.java:1258', 'org.apache.thrift.ProcessFunction:process:ProcessFunction.java:39', 'org.apache.thrift.TBaseProcessor:process:TBaseProcessor.java:39', 'org.apache.hive.service.auth.TSetIpAddressProcessor:process:TSetIpAddressProcessor.java:56', 'org.apache.thrift.server.TThreadPoolServer$WorkerProcess:run:TThreadPoolServer.java:285', 'java.util.concurrent.ThreadPoolExecutor:runWorker:ThreadPoolExecutor.java:1145', 'java.util.concurrent.ThreadPoolExecutor$Worker:run:ThreadPoolExecutor.java:615', 'java.lang.Thread:run:Thread.java:745'], statusCode=3))

Issue Links

• relates to: HIVE-14067 Rename pendingCount to activeCalls in HiveSessionImpl for easier understanding.