Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-9345

Hang in TestOrc::test_type_conversions when running create table in hive

Agile BoardAttach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Epic Color:
      ghx-label-4

      Description

      query_test.test_scanners.TestOrc.test_type_conversions[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: orc/def/block] (from pytest)
      Failing for the past 1 build (Since Failed#93 )
      Took 2 hr 0 min.
      add description
      Error Message
      
      query_test/test_scanners.py:1337: in test_type_conversions     self.run_test_case('DataErrorsTest/orc-type-checks', vector, unique_database) common/impala_test_suite.py:659: in run_test_case     result = exec_fn(query, user=test_section.get('USER', '').strip() or None) common/impala_test_suite.py:610: in __exec_in_hive     result = h.execute(query, user=user) common/impala_connection.py:334: in execute     r = self.__fetch_results(handle, profile_format=profile_format) common/impala_connection.py:441: in __fetch_results     cursor._wait_to_finish() /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/hiveserver2.py:405: in _wait_to_finish     resp = self._last_operation._rpc('GetOperationStatus', req) /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/hiveserver2.py:992: in _rpc     response = self._execute(func_name, request) /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/hiveserver2.py:1009: in _execute     return func(request) /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/_thrift_gen/TCLIService/TCLIService.py:517: in GetOperationStatus     return self.recv_GetOperationStatus() /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/_thrift_gen/TCLIService/TCLIService.py:529: in recv_GetOperationStatus     (fname, mtype, rseqid) = iprot.readMessageBegin() /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/Impala-Toolchain/thrift-0.9.3-p7/python/lib64/python2.7/site-packages/thrift/protocol/TBinaryProtocol.py:126: in readMessageBegin     sz = self.readI32() /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/Impala-Toolchain/thrift-0.9.3-p7/python/lib64/python2.7/site-packages/thrift/protocol/TBinaryProtocol.py:206: in readI32     buff = self.trans.readAll(4) /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/Impala-Toolchain/thrift-0.9.3-p7/python/lib64/python2.7/site-packages/thrift/transport/TTransport.py:58: in readAll     chunk = self.read(sz - have) /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/thrift_sasl/__init__.py:159: in read     self._read_frame() /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/thrift_sasl/__init__.py:163: in _read_frame     header = read_all_compat(self._trans, 4) /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/thrift_sasl/six.py:31: in <lambda>     read_all_compat = lambda trans, sz: trans.readAll(sz) /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/Impala-Toolchain/thrift-0.9.3-p7/python/lib64/python2.7/site-packages/thrift/transport/TTransport.py:58: in readAll     chunk = self.read(sz - have) /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/Impala-Toolchain/thrift-0.9.3-p7/python/lib64/python2.7/site-packages/thrift/transport/TSocket.py:105: in read     buff = self.handle.recv(sz) E   Failed: Timeout >7200s
      
      Stacktrace
      
      query_test/test_scanners.py:1337: in test_type_conversions
          self.run_test_case('DataErrorsTest/orc-type-checks', vector, unique_database)
      common/impala_test_suite.py:659: in run_test_case
          result = exec_fn(query, user=test_section.get('USER', '').strip() or None)
      common/impala_test_suite.py:610: in __exec_in_hive
          result = h.execute(query, user=user)
      common/impala_connection.py:334: in execute
          r = self.__fetch_results(handle, profile_format=profile_format)
      common/impala_connection.py:441: in __fetch_results
          cursor._wait_to_finish()
      /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/hiveserver2.py:405: in _wait_to_finish
          resp = self._last_operation._rpc('GetOperationStatus', req)
      /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/hiveserver2.py:992: in _rpc
          response = self._execute(func_name, request)
      /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/hiveserver2.py:1009: in _execute
          return func(request)
      /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/_thrift_gen/TCLIService/TCLIService.py:517: in GetOperationStatus
          return self.recv_GetOperationStatus()
      /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/_thrift_gen/TCLIService/TCLIService.py:529: in recv_GetOperationStatus
          (fname, mtype, rseqid) = iprot.readMessageBegin()
      /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/Impala-Toolchain/thrift-0.9.3-p7/python/lib64/python2.7/site-packages/thrift/protocol/TBinaryProtocol.py:126: in readMessageBegin
          sz = self.readI32()
      /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/Impala-Toolchain/thrift-0.9.3-p7/python/lib64/python2.7/site-packages/thrift/protocol/TBinaryProtocol.py:206: in readI32
          buff = self.trans.readAll(4)
      /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/Impala-Toolchain/thrift-0.9.3-p7/python/lib64/python2.7/site-packages/thrift/transport/TTransport.py:58: in readAll
          chunk = self.read(sz - have)
      /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/thrift_sasl/__init__.py:159: in read
          self._read_frame()
      /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/thrift_sasl/__init__.py:163: in _read_frame
          header = read_all_compat(self._trans, 4)
      /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/thrift_sasl/six.py:31: in <lambda>
          read_all_compat = lambda trans, sz: trans.readAll(sz)
      /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/Impala-Toolchain/thrift-0.9.3-p7/python/lib64/python2.7/site-packages/thrift/transport/TTransport.py:58: in readAll
          chunk = self.read(sz - have)
      /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/Impala-Toolchain/thrift-0.9.3-p7/python/lib64/python2.7/site-packages/thrift/transport/TSocket.py:105: in read
          buff = self.handle.recv(sz)
      E   Failed: Timeout >7200s
      

      This has happened on multiple builds, all running against S3. It seems to be happening with high frequency on S3, but isn't deterministic - I saw this commit both succeed and fail:

          IMPALA-9334: Log node blacklisting removals in ClusterMembershipMgr
      

      All failures did occur after " IMPALA-9242: Bump CDH_BUILD_NUMBER to 1814051" but not sure if that's related or not.

      Sample hive log:

      tarmstrong@tarmstrong-box:~/tmplogs/impala-asf-master-core-s3-data-cache-93/logs/cluster$ grep a6c81474e8c hive*
      grep: hive: Is a directory
      hive-server2.log:2020-01-28T11:10:49,558  INFO [e00c5c82-1a05-4fc7-bed2-1c62751c2223 HiveServer2-Handler-Pool: Thread-41] ql.Driver: Compiling command(queryId=jenkins_20200128111049_91b0e648-7740-431f-88bb-a6c81474e8c9): create external table test_type_conversions_3e81315e.union_complextypes(
      hive-server2.log:2020-01-28T11:10:50,798  INFO [e00c5c82-1a05-4fc7-bed2-1c62751c2223 HiveServer2-Handler-Pool: Thread-41] ql.Driver: Completed compiling command(queryId=jenkins_20200128111049_91b0e648-7740-431f-88bb-a6c81474e8c9); Time taken: 1.274 seconds
      hive-server2.log:2020-01-28T13:10:16,567  INFO [e00c5c82-1a05-4fc7-bed2-1c62751c2223 HiveServer2-Handler-Pool: Thread-41] operation.Operation: The running operation has been successfully interrupted: jenkins_20200128111049_91b0e648-7740-431f-88bb-a6c81474e8c9
      

        Attachments

        Issue Links

          Activity

            People

            • Assignee:
              tarmstrong Tim Armstrong
              Reporter:
              tarmstrong Tim Armstrong

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment