[IMPALA-4228] Impala JDBC: Concurrent connections - ResultSet#next() locks execution when no more records in ResultSet

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Not A Bug
• Affects Version/s: Impala 2.3.0
• Component/s: Perf Investigation

Environment

#hadoop
$ hadoop version
Hadoop 2.6.0-cdh5.5.4

#impala
> version;
Shell version: Impala Shell v2.3.0-cdh5.5.4 (e65ded2) built on Mon Apr 25 10:55:11 PDT 2016
Server version: impalad version 2.3.0-cdh5.5.4 RELEASE (build e65ded24350974ae6b4e475557b358b718fad29e)

#os
$ lsb_release -d;
Description: Red Hat Enterprise Linux Server release 7.2 (Maipo)
$ uname -a
Linux gcphxd11.uk.db.com 3.10.0-327.28.2.el7.x86_64 #1 SMP Mon Jun 27 14:48:28 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux

#JDBC driver
Cloudera_ImpalaJDBC41_2.5.32

• Assignee: Unassigned
• Reporter: Valeriy Cherepanov
• Created: 29/Sep/16 16:04
• Updated: 27/Oct/16 07:41
• Resolved: 27/Oct/16 07:40

Description

While running performance tests I'm noticing that quite often execution of all threads are blocked around invocation of ResultSet#next() when there are no more records in result set and ResultSet#next() will eventually return false.
Particularly in my test I'm periodically starting 10 connections with each connection to run the same query 10 times - i.e. 10*10=100 queries for every run.
The query is 'select * from <some_table> limit 10'
The issue doesn't take place in single-threaded mode, only with multiple connections working simultaneously.
See related log lines generated by the attached reproduction code:
16:33:24.781 1304: BEFORE rs.next()
16:33:24.781 1304: AFTER rs.next() next = true
16:33:24.781 1304: BEFORE rs.next()
16:33:24.781 1304: AFTER rs.next() next = true
16:33:24.781 1304: BEFORE rs.next() //this is the last rs.next() for cycle no. 1304
//nothing here for 25 second, looks like locking issue, notice that several threads are calling ResultSet.next for the last time
16:33:50.049 1312: AFTER rs.next() next = false
16:33:50.049 1309: AFTER rs.next() next = false
16:33:50.049 1309: AFTER handling ResultSet
16:33:50.049 1307: AFTER rs.next() next = false
16:33:50.049 1308: AFTER rs.next() next = false
16:33:50.049 1314: BEFORE stmt.executeQuery
16:33:50.049 1304: AFTER rs.next() next = false //rs.next() for cycle no. 1304 returned in 25 seconds
16:33:50.049 1313: AFTER rs.next() next = false
16:33:50.049 1313: AFTER handling ResultSet
16:33:50.049 1304: AFTER handling ResultSet