[IMPALA-7796] TestAutomaticCatalogInvalidation custom cluster suite failing for both local and V1

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Duplicate
• Affects Version/s: Impala 3.1.0
• Component/s: Frontend
• Epic Color: ghx-label-2
• Assignee: Vuk Ercegovac
• Reporter: David Knupp
• Created: 31/Oct/18 21:38
• Updated: 31/Oct/18 21:40
• Resolved: 31/Oct/18 21:40

Description

Both variants exceed max wait time.

v1 catalog:

custom_cluster/test_automatic_invalidation.py:69: in test_v1_catalog
    self._run_test(cursor)
custom_cluster/test_automatic_invalidation.py:64: in _run_test
    assert time.time() < max_wait_time
E   assert 1541000646.910642 < 1541000646.673253
E    +  where 1541000646.910642 = <built-in function time>()
E    +    where <built-in function time> = time.time

Local catalog

custom_cluster/test_automatic_invalidation.py:76: in test_local_catalog
    self._run_test(cursor)
custom_cluster/test_automatic_invalidation.py:64: in _run_test
    assert time.time() < max_wait_time
E   assert 1541000679.388713 < 1541000679.148656
E    +  where 1541000679.388713 = <built-in function time>()
E    +    where <built-in function time> = time.time

Additionally, the v1 catalog test seemed to experience some connectivity issues:

-- 2018-10-31 08:44:18,118 INFO     MainThread: num_known_live_backends has reached value: 3
-- connecting to: localhost:21000
-- connecting to localhost:21050 with impyla
Conn <impala.hiveserver2.HiveServer2Connection object at 0x7f6a80ab57d0>
-- 2018-10-31 08:44:18,214 INFO     MainThread: Closing active operation
-- 2018-10-31 08:44:18,215 ERROR    MainThread: Failed to open transport (tries_left=3)
Traceback (most recent call last):
  File "/data/jenkins/workspace/impala-asf-master-exhaustive/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/hiveserver2.py", line 940, in _execute
    return func(request)
  File "/data/jenkins/workspace/impala-asf-master-exhaustive/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/_thrift_gen/TCLIService/TCLIService.py", line 175, in OpenSession
    return self.recv_OpenSession()
  File "/data/jenkins/workspace/impala-asf-master-exhaustive/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/_thrift_gen/TCLIService/TCLIService.py", line 186, in recv_OpenSession
    (fname, mtype, rseqid) = self._iprot.readMessageBegin()
  File "/data/jenkins/workspace/impala-asf-master-exhaustive/Impala-Toolchain/thrift-0.9.3-p4/python/lib64/python2.7/site-packages/thrift/protocol/TBinaryProtocol.py", line 126, in readMessageBegin
    sz = self.readI32()
  File "/data/jenkins/workspace/impala-asf-master-exhaustive/Impala-Toolchain/thrift-0.9.3-p4/python/lib64/python2.7/site-packages/thrift/protocol/TBinaryProtocol.py", line 206, in readI32
    buff = self.trans.readAll(4)
  File "/data/jenkins/workspace/impala-asf-master-exhaustive/Impala-Toolchain/thrift-0.9.3-p4/python/lib64/python2.7/site-packages/thrift/transport/TTransport.py", line 58, in readAll
    chunk = self.read(sz - have)
  File "/data/jenkins/workspace/impala-asf-master-exhaustive/Impala-Toolchain/thrift-0.9.3-p4/python/lib64/python2.7/site-packages/thrift/transport/TTransport.py", line 159, in read
    self.__rbuf = StringIO(self.__trans.read(max(sz, self.__rbuf_size)))
  File "/data/jenkins/workspace/impala-asf-master-exhaustive/Impala-Toolchain/thrift-0.9.3-p4/python/lib64/python2.7/site-packages/thrift/transport/TSocket.py", line 120, in read
    message='TSocket read 0 bytes')
TTransportException: TSocket read 0 bytes
-- 2018-10-31 08:44:19,133 INFO     MainThread: Starting new HTTP connection (1): impala-ec2-centos74-m5-4xlarge-ondemand-02bb.vpc.cloudera.com
-- 2018-10-31 08:44:20,150 INFO     MainThread: Starting new HTTP connection (1): impala-ec2-centos74-m5-4xlarge-ondemand-02bb.vpc.cloudera.com

Issue Links

• duplicates: IMPALA-7594 TestAutomaticCatalogInvalidation.test_local_catalog and test_v1_catalog still hitting timeout