[HIVE-19117] hiveserver2 org.apache.thrift.transport.TTransportException error when running 2nd query after minute of inactivity

• Type: Bug
• Status: Open
• Priority: Blocker
• Resolution: Unresolved
• Affects Version/s: 2.1.1
• Component/s: Hive, HiveServer2, Metastore, Thrift API

Environment

• Hive 2.1.1 with hive.server2.transport.mode set to binary (sample JDBC string is jdbc:hive2://remotehost:10000/default)
• Hadoop 2.8.3
• Metastore using MySQL
• Java 8

• Assignee: Unassigned
• Reporter: t oo
• Created: 05/Apr/18 11:07
• Updated: 21/Feb/19 17:32

Description

I make a JDBC connection from my SQL tool (ie Squirrel SQL, Oracle SQL Developer) to HiveServer2 (running on remote server) with port 10000.

I am able to run some queries successfully. I then do something else (not in the SQL tool) for 1-2minutes and then return to my SQL tool and attempt to run a query but I get this error:

org.apache.thrift.transport.TTransportException: java.net.SocketException: Software caused connection abort: socket write error

If I now disconnect and reconnect in my SQL tool I can run queries again. But does anyone know what HiveServer2 settings I should change to prevent the error? I assume something in hive-site.xml

From the hiveserver2 logs below, can see an exact 1 minute gap from 30th min to 31stmin where the disconnect happens.

2018-04-05T03:30:41,706 INFO [HiveServer2-Handler-Pool: Thread-36] session.SessionState: Resetting thread name to HiveServer2-Handler-Pool: Thread-36
 2018-04-05T03:30:41,712 INFO [HiveServer2-Handler-Pool: Thread-36] session.SessionState: Updating thread name to c81ec0f9-7a9d-46b6-9708-e7d78520a48a HiveServer2-Handler-Pool: Thread-36
 2018-04-05T03:30:41,712 INFO [HiveServer2-Handler-Pool: Thread-36] session.SessionState: Resetting thread name to HiveServer2-Handler-Pool: Thread-36
 2018-04-05T03:30:41,718 INFO [HiveServer2-Handler-Pool: Thread-36] session.SessionState: Updating thread name to c81ec0f9-7a9d-46b6-9708-e7d78520a48a HiveServer2-Handler-Pool: Thread-36
 2018-04-05T03:30:41,719 INFO [HiveServer2-Handler-Pool: Thread-36] session.SessionState: Resetting thread name to HiveServer2-Handler-Pool: Thread-36
 2018-04-05T03:31:41,232 INFO [HiveServer2-Handler-Pool: Thread-36] thrift.ThriftCLIService: Session disconnected without closing properly.
 2018-04-05T03:31:41,233 INFO [HiveServer2-Handler-Pool: Thread-36] thrift.ThriftCLIService: Closing the session: SessionHandle [c81ec0f9-7a9d-46b6-9708-e7d78520a48a]
 2018-04-05T03:31:41,233 INFO [HiveServer2-Handler-Pool: Thread-36] service.CompositeService: Session closed, SessionHandle [c81ec0f9-7a9d-46b6-9708-e7d78520a48a], current sessions:0
 2018-04-05T03:31:41,233 INFO [HiveServer2-Handler-Pool: Thread-36] session.SessionState: Updating thread name to c81ec0f9-7a9d-46b6-9708-e7d78520a48a HiveServer2-Handler-Pool: Thread-36
 2018-04-05T03:31:41,233 INFO [HiveServer2-Handler-Pool: Thread-36] session.SessionState: Resetting thread name to HiveServer2-Handler-Pool: Thread-36
 2018-04-05T03:31:41,233 INFO [HiveServer2-Handler-Pool: Thread-36] session.SessionState: Updating thread name to c81ec0f9-7a9d-46b6-9708-e7d78520a48a HiveServer2-Handler-Pool: Thread-36
 2018-04-05T03:31:41,233 INFO [HiveServer2-Handler-Pool: Thread-36] session.HiveSessionImpl: Operation log session directory is deleted: /var/hive/hs2log/tmp/c81ec0f9-7a9d-46b6-9708-e7d78520a48a
 2018-04-05T03:31:41,233 INFO [HiveServer2-Handler-Pool: Thread-36] session.SessionState: Resetting thread name to HiveServer2-Handler-Pool: Thread-36
 2018-04-05T03:31:41,236 INFO [HiveServer2-Handler-Pool: Thread-36] session.SessionState: Deleted directory: /var/hive/scratch/tmp/anonymous/c81ec0f9-7a9d-46b6-9708-e7d78520a48a on fs with scheme file
 2018-04-05T03:31:41,236 INFO [HiveServer2-Handler-Pool: Thread-36] session.SessionState: Deleted directory: /var/hive/ec2-user/c81ec0f9-7a9d-46b6-9708-e7d78520a48a on fs with scheme file
 2018-04-05T03:31:41,236 INFO [HiveServer2-Handler-Pool: Thread-36] hive.metastore: Closed a connection to metastore, current connections: 1