[IMPALA-7238] test_kudu.TestCreateExternalTable sees unique database already exists

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 3.1.0
• Fix Version/s: Impala 2.13.0, Impala 3.1.0
• Component/s: Infrastructure
• Labels: broken-build flaky
• Target Version: Impala 3.1.0
• Epic Color: ghx-label-3
• Assignee: Joe McDonnell
• Reporter: Joe McDonnell
• Created: 03/Jul/18 00:05
• Updated: 29/May/19 15:34
• Resolved: 17/Jul/18 16:55

Description

All of the tests from query_test.test_kudu.TestCreateExternalTable fail with an error like:

/data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/hiveserver2.py:704: in err_if_rpc_not_ok
    raise HiveServer2Error(resp.status.errorMessage)
E   HiveServer2Error: ImpalaRuntimeException: Error making 'createDatabase' RPC to Hive Metastore: 
E   CAUSED BY: AlreadyExistsException: Database testcreateexternaltable_23808_vu8cqo already exists

It looks like the failures all happen at once in a single process. The first test to fail is test_kudu.TestCreateExternalTable.test_col_types. It takes 52 seconds where all the other tests take no time. It also has an extra error on stderr:

-- connecting to: localhost:21000
MainThread: Failed to open transport (tries_left=3)
Traceback (most recent call last):
  File "/data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/hiveserver2.py", line 940, in _execute
    return func(request)
  File "/data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/_thrift_gen/TCLIService/TCLIService.py", line 265, in ExecuteStatement
    return self.recv_ExecuteStatement()
  File "/data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/_thrift_gen/TCLIService/TCLIService.py", line 276, in recv_ExecuteStatement
    (fname, mtype, rseqid) = self._iprot.readMessageBegin()
  File "/data/jenkins/workspace/impala-asf-master-core-s3/Impala-Toolchain/thrift-0.9.3-p4/python/lib64/python2.7/site-packages/thrift/protocol/TBinaryProtocol.py", line 126, in readMessageBegin
    sz = self.readI32()
  File "/data/jenkins/workspace/impala-asf-master-core-s3/Impala-Toolchain/thrift-0.9.3-p4/python/lib64/python2.7/site-packages/thrift/protocol/TBinaryProtocol.py", line 206, in readI32
    buff = self.trans.readAll(4)
  File "/data/jenkins/workspace/impala-asf-master-core-s3/Impala-Toolchain/thrift-0.9.3-p4/python/lib64/python2.7/site-packages/thrift/transport/TTransport.py", line 58, in readAll
    chunk = self.read(sz - have)
  File "/data/jenkins/workspace/impala-asf-master-core-s3/Impala-Toolchain/thrift-0.9.3-p4/python/lib64/python2.7/site-packages/thrift/transport/TTransport.py", line 159, in read
    self.__rbuf = StringIO(self.__trans.read(max(sz, self.__rbuf_size)))
  File "/data/jenkins/workspace/impala-asf-master-core-s3/Impala-Toolchain/thrift-0.9.3-p4/python/lib64/python2.7/site-packages/thrift/transport/TSocket.py", line 105, in read
    buff = self.handle.recv(sz)
timeout: timed out
MainThread: Error closing Impala cursor: Invalid session id: f54064f9a4604f23:fb686144269fc8b1

The other failures don't have this.

This happened only once, so it is definitely intermittent. This has some similarity to IMPALA-6933, but this looks like a repeated failure in a single process, not a concurrency issue.