[IMPALA-8603] Some custom cluster tests print error messages after IMPALA-1653 was fixed

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 3.3.0
• Fix Version/s: Impala 3.3.0
• Component/s: Infrastructure
• Labels: hs2 regression
• Epic Color: ghx-label-7
• Assignee: Thomas Tauber-Marshall
• Reporter: Lars Volker
• Created: 30/May/19 20:13
• Updated: 10/Jun/19 16:31
• Resolved: 10/Jun/19 16:31

Description

The fix for IMPALA-1653 seems to introduce error messages at the end of some custom cluster tests. It looks like tests are affected that leave the cluster down.

PASSED-- closing connection to: localhost:21000                                                                                                                                                      [4/768]
-- closing connection to: localhost:21050
-- 2019-05-30 13:02:56,968 ERROR    MainThread: Failed to open transport (tries_left=3)
Traceback (most recent call last):
  File "/home/lv/i7/infra/python/env/local/lib/python2.7/site-packages/impala/hiveserver2.py", line 961, in _execute
    return func(request)
  File "/home/lv/i7/infra/python/env/local/lib/python2.7/site-packages/impala/_thrift_gen/TCLIService/TCLIService.py", line 207, in CloseSession
    return self.recv_CloseSession()
  File "/home/lv/i7/infra/python/env/local/lib/python2.7/site-packages/impala/_thrift_gen/TCLIService/TCLIService.py", line 219, in recv_CloseSession
    (fname, mtype, rseqid) = iprot.readMessageBegin()
  File "/opt/Impala-Toolchain/thrift-0.9.3-p5/python/lib/python2.7/site-packages/thrift/protocol/TBinaryProtocol.py", line 126, in readMessageBegin
    sz = self.readI32()
  File "/opt/Impala-Toolchain/thrift-0.9.3-p5/python/lib/python2.7/site-packages/thrift/protocol/TBinaryProtocol.py", line 206, in readI32
    buff = self.trans.readAll(4)
  File "/opt/Impala-Toolchain/thrift-0.9.3-p5/python/lib/python2.7/site-packages/thrift/transport/TTransport.py", line 58, in readAll
    chunk = self.read(sz - have)
  File "/opt/Impala-Toolchain/thrift-0.9.3-p5/python/lib/python2.7/site-packages/thrift/transport/TTransport.py", line 159, in read
    self.__rbuf = StringIO(self.__trans.read(max(sz, self.__rbuf_size)))
  File "/opt/Impala-Toolchain/thrift-0.9.3-p5/python/lib/python2.7/site-packages/thrift/transport/TSocket.py", line 120, in read
    message='TSocket read 0 bytes')
TTransportException: TSocket read 0 bytes
-- 2019-05-30 13:02:56,969 ERROR    MainThread: Failed to open transport (tries_left=2)
Traceback (most recent call last):
  File "/home/lv/i7/infra/python/env/local/lib/python2.7/site-packages/impala/hiveserver2.py", line 958, in _execute
    open_transport(transport)
  File "/home/lv/i7/infra/python/env/local/lib/python2.7/site-packages/impala/hiveserver2.py", line 990, in open_transport
    transport.open()
  File "/opt/Impala-Toolchain/thrift-0.9.3-p5/python/lib/python2.7/site-packages/thrift/transport/TTransport.py", line 149, in open
    return self.__trans.open()
  File "/opt/Impala-Toolchain/thrift-0.9.3-p5/python/lib/python2.7/site-packages/thrift/transport/TSocket.py", line 101, in open
    message=message)
TTransportException: Could not connect to localhost:21050
-- 2019-05-30 13:02:56,970 ERROR    MainThread: Failed to open transport (tries_left=1)
Traceback (most recent call last):
  File "/home/lv/i7/infra/python/env/local/lib/python2.7/site-packages/impala/hiveserver2.py", line 958, in _execute
    open_transport(transport)
  File "/home/lv/i7/infra/python/env/local/lib/python2.7/site-packages/impala/hiveserver2.py", line 990, in open_transport
    transport.open()
  File "/opt/Impala-Toolchain/thrift-0.9.3-p5/python/lib/python2.7/site-packages/thrift/transport/TTransport.py", line 149, in open
    return self.__trans.open()
  File "/opt/Impala-Toolchain/thrift-0.9.3-p5/python/lib/python2.7/site-packages/thrift/transport/TSocket.py", line 101, in open
    message=message)
TTransportException: Could not connect to localhost:21050
Failed after retrying 3 times
Traceback (most recent call last):
  File "/home/lv/i7/tests/common/impala_connection.py", line 283, in close
    self.__cursor.close()
  File "/home/lv/i7/infra/python/env/local/lib/python2.7/site-packages/impala/hiveserver2.py", line 248, in close
    self.session.close()
  File "/home/lv/i7/infra/python/env/local/lib/python2.7/site-packages/impala/hiveserver2.py", line 1041, in close
    self._rpc('CloseSession', req)
  File "/home/lv/i7/infra/python/env/local/lib/python2.7/site-packages/impala/hiveserver2.py", line 944, in _rpc
    response = self._execute(func_name, request)
  File "/home/lv/i7/infra/python/env/local/lib/python2.7/site-packages/impala/hiveserver2.py", line 975, in _execute
    .format(self.retries))
HiveServer2Error: Failed after retrying 3 times

The tests still return 0 (success) but I found this confusing.

twmarshall - I'm not sure what the best workaround here is. Should we ignore "Could not connect to" in impala_connection.py.