[HIVE-25153] JDBC connection randomly fails if both leader and zookeeper discovery namespaces are within znode /hiveserver2

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.2
• Component/s: HiveServer2

Environment

Hive 3.1.2

• Assignee: Unassigned
• Reporter: Pranay
• Created: 21/May/21 22:24
• Updated: 10/Aug/21 01:47

Description

Hello, 

jdbc connection randomly fails to connect to hiveserver2.  Here are observation - 

Zookeeper znodes

[zk: redacted106.visa.com:2181(CONNECTED) 5] ls /hiveserver2
[leader, serverUri=redacted206.visa.com:10000;version=3.1.2-1-SNAPSHOT;sequence=0000000007]

 

ZooKeeperHiveClientHelper.java randomly reads server list from the znode from /hiveserver2 and when it reads leader as random host, it fails to connect. 

Failed jdbc connection
------------------------

21/05/21 20:14:17 [main-SendThread(redacted106.visa.com:2181)]: DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x102e58cc0bd0019, packet:: clientPath:null serverPath:null finished:false header:: 1,12 replyHeader:: 1,42949673107,0 request:: '/hiveserver2,F response:: v{'leader,'serverUri=redacted206.visa.com:10000;version=3.1.2-1-SNAPSHOT;sequence=0000000005},s{21474836584,21474836584,1621616800534,1621616800534,0,10,0,0,13,2,42949673008}
21/05/21 20:14:17 [main-SendThread(redacted106.visa.com:2181)]: DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x102e58cc0bd0019, packet:: clientPath:null serverPath:null finished:false header:: 2,4 replyHeader:: 2,42949673107,0 request:: '/hiveserver2/leader,F response:: ,s{21474836589,21474836589,1621616800587,1621616800587,0,5,0,0,0,1,42949673013}
2
21/05/21 20:14:17 [main-SendThread(redacted106.visa.com:2181)]: DEBUG zookeeper.ClientCnxn: An exception was thrown while closing send thread for session 0x102e58cc0bd0019 : Unable to read additional data from server sessionid 0x102e58cc0bd0019, likely server has closed socket
Error: org.apache.hive.jdbc.ZooKeeperHiveClientException: Unable to read HiveServer2 configs from ZooKeeper (state=,code=0)

 

 

Successful jdbc connection
----------------------------

 

21/05/21 20:14:01 [main-SendThread(redacted106.visa.com:2181)]: DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x102e58cc0bd0018, packet:: clientPath:null serverPath:null finished:false header:: 1,12 replyHeader:: 1,42949673103,0 request:: '/hiveserver2,F response:: v{'leader,'serverUri=redacted206.visa.com:10000;version=3.1.2-1-SNAPSHOT;sequence=0000000005},s{21474836584,21474836584,1621616800534,1621616800534,0,10,0,0,13,2,42949673008}
21/05/21 20:14:01 [main-SendThread(redacted106.visa.com:2181)]: DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x102e58cc0bd0018, packet:: clientPath:null serverPath:null finished:false header:: 2,4 replyHeader:: 2,42949673103,0 request:: '/hiveserver2/serverUri=redacted206.visa.com:10000;version=3.1.2-1-SNAPSHOT;sequence=0000000005,F response:: #686976652e736572766572322e696e7374616e63652e7572693d736c373364706968636e30303230362e766973612e636f6d3a31303030303b686976652e736572766572322e61757468656e7469636174696f6e3d4b45524245524f533b686976652e736572766572322e7472616e73706f72742e6d6f64653d62696e6172793b686976652e736572766572322e7468726966742e7361736c2e716f703d617574683b686976652e736572766572322e7468726966742e62696e642e686f73743d736c373364706968636e30303230362e766973612e636f6d3b686976652e736572766572322e7468726966742e706f72743d31303030303b686976652e736572766572322e7573652e53534c3d66616c73653b686976652e736572766572322e61757468656e7469636174696f6e2e6b65726265726f732e7072696e636970616c3d686976652f5f484f535440434f52504445562e564953412e434f4d,s{42949673008,42949673008,1621626552349,1621626552349,0,0,0,72872936683143172,350,0,42949673008}
21/05/21 20:14:01 [main]: DEBUG jdbc.Utils: Resolved authority: redacted206.visa.com:10000

 

 

Issue Links

• duplicates: HIVE-19825 HiveServer2 leader selection shall use different zookeeper znode