Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-5816

ssl-related custom cluster tests failing during setup on exhaustive RHEL7

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: Impala 2.10.0
    • Fix Version/s: Impala 2.11.0
    • Component/s: Security
    • Labels:
      None
    • Epic Color:
      ghx-label-3

      Description

      Tests that were seen to fail include:

      • TestClientSsl.test_tls_v12
      • TestClientSsl.test_wildcard_ssl
      • TestClientSsl.test_wildcard_san_ssl

      Example stack trace:

      self = <test_client_ssl.TestClientSsl object at 0x5bf1090>
      method = <bound method TestClientSsl.test_tls_v12 of <test_client_ssl.TestClientSsl object at 0x5bf1090>>
      
          def setup_method(self, method):
            cluster_args = list()
            for arg in [IMPALAD_ARGS, STATESTORED_ARGS, CATALOGD_ARGS]:
              if arg in method.func_dict:
                cluster_args.append("--%s=\"%s\" " % (arg, method.func_dict[arg]))
            # Start a clean new cluster before each test
      >     self._start_impala_cluster(cluster_args)
      
      common/custom_cluster_test_suite.py:103: 
      _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
      common/custom_cluster_test_suite.py:129: in _start_impala_cluster
          check_call(cmd + options, close_fds=True)
      _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
      
      popenargs = (['/data/jenkins/workspace/impala-cdh5-trunk-exhaustive-rhel7/repos/Impala/bin/start-impala-cluster.py', '--cluster_si...cdh5-trunk-exhaustive-rhel7/repos/Impala/be/src/testutil/wildcardCA.pem --ssl_cipher_list=AES128-GCM-SHA256 " ', ...],)
      kwargs = {'close_fds': True}, retcode = 1
      cmd = ['/data/jenkins/workspace/impala-cdh5-trunk-exhaustive-rhel7/repos/Impala/bin/start-impala-cluster.py', '--cluster_siz...a-cdh5-trunk-exhaustive-rhel7/repos/Impala/be/src/testutil/wildcardCA.pem --ssl_cipher_list=AES128-GCM-SHA256 " ', ...]
      
          def check_call(*popenargs, **kwargs):
              """Run command with arguments.  Wait for command to complete.  If
              the exit code was zero then return, otherwise raise
              CalledProcessError.  The CalledProcessError object will have the
              return code in the returncode attribute.
          
              The arguments are the same as for the Popen constructor.  Example:
          
              check_call(["ls", "-l"])
              """
              retcode = call(*popenargs, **kwargs)
              if retcode:
                  cmd = kwargs.get("args")
                  if cmd is None:
                      cmd = popenargs[0]
      >           raise CalledProcessError(retcode, cmd)
      E           CalledProcessError: Command '['/data/jenkins/workspace/impala-cdh5-trunk-exhaustive-rhel7/repos/Impala/bin/start-impala-cluster.py', '--cluster_size=3', '--num_coordinators=3', '--log_dir=/data/jenkins/workspace/impala-cdh5-trunk-exhaustive-rhel7/repos/Impala/logs/custom_cluster_tests', '--log_level=1', '--impalad_args="--ssl_server_certificate=/data/jenkins/workspace/impala-cdh5-trunk-exhaustive-rhel7/repos/Impala/be/src/testutil/wildcard-cert.pem --ssl_private_key=/data/jenkins/workspace/impala-cdh5-trunk-exhaustive-rhel7/repos/Impala/be/src/testutil/wildcard-cert.key --ssl_minimum_version=tlsv1.2 --ssl_client_ca_certificate=/data/jenkins/workspace/impala-cdh5-trunk-exhaustive-rhel7/repos/Impala/be/src/testutil/wildcardCA.pem --ssl_cipher_list=AES128-GCM-SHA256 " ', '--state_store_args="--ssl_server_certificate=/data/jenkins/workspace/impala-cdh5-trunk-exhaustive-rhel7/repos/Impala/be/src/testutil/wildcard-cert.pem --ssl_private_key=/data/jenkins/workspace/impala-cdh5-trunk-exhaustive-rhel7/repos/Impala/be/src/testutil/wildcard-cert.key --ssl_minimum_version=tlsv1.2 --ssl_client_ca_certificate=/data/jenkins/workspace/impala-cdh5-trunk-exhaustive-rhel7/repos/Impala/be/src/testutil/wildcardCA.pem --ssl_cipher_list=AES128-GCM-SHA256 " ', '--catalogd_args="--ssl_server_certificate=/data/jenkins/workspace/impala-cdh5-trunk-exhaustive-rhel7/repos/Impala/be/src/testutil/wildcard-cert.pem --ssl_private_key=/data/jenkins/workspace/impala-cdh5-trunk-exhaustive-rhel7/repos/Impala/be/src/testutil/wildcard-cert.key --ssl_minimum_version=tlsv1.2 --ssl_client_ca_certificate=/data/jenkins/workspace/impala-cdh5-trunk-exhaustive-rhel7/repos/Impala/be/src/testutil/wildcardCA.pem --ssl_cipher_list=AES128-GCM-SHA256 " ']' returned non-zero exit status 1
      

      Standard error output:

      MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
      MainThread: Getting num_known_live_backends from impala-boost-static-burst-slave-el7-1b82.vpc.cloudera.com:25000
      MainThread: Debug webpage not yet available.
      MainThread: Waiting for num_known_live_backends=3. Current value: 0
      MainThread: Getting num_known_live_backends from impala-boost-static-burst-slave-el7-1b82.vpc.cloudera.com:25000
      MainThread: Waiting for num_known_live_backends=3. Current value: 0
      MainThread: Getting num_known_live_backends from impala-boost-static-burst-slave-el7-1b82.vpc.cloudera.com:25000
      MainThread: Waiting for num_known_live_backends=3. Current value: 0
      MainThread: Getting num_known_live_backends from impala-boost-static-burst-slave-el7-1b82.vpc.cloudera.com:25000
      MainThread: Waiting for num_known_live_backends=3. Current value: 0
      etc...
      etc...
      

      When I look at the impalad.INFO log, this entry shows up at that timestamp:

      I0818 00:43:29.000936 17712 Frontend.java:888] Compiling query: drop role grant_revoke_test_admin
      I0818 00:43:29.001284 17712 Frontend.java:927] Compiled query.
      I0818 00:43:29.021884 17712 impala-beeswax-server.cc:189] get_results_metadata(): query_id=114d057e65f76f3d:60e4c38f00000000
      I0818 00:43:29.316861 17712 status.cc:122] Query id 114d057e65f76f3d:60e4c38f00000000 not found.
          @          0x12574e3  impala::Status::Status()
          @          0x1512d22  impala::ImpalaServer::GetExecSummary()
          @          0x158594a  impala::ImpalaServer::GetExecSummary()
          @          0x1a595d5  impala::ImpalaServiceProcessor::process_GetExecSummary()
          @          0x1a56619  impala::ImpalaServiceProcessor::dispatchCall()
          @          0x11fbb96  apache::thrift::TDispatchProcessor::process()
          @          0x29076db  apache::thrift::server::TThreadPoolServer::Task::run()
          @          0x28efed9  apache::thrift::concurrency::ThreadManager::Worker::run()
          @          0x13c7a6f  impala::ThriftThread::RunRunnable()
          @          0x13c919b  boost::_mfi::mf2<>::operator()()
          @          0x13c9031  boost::_bi::list3<>::operator()<>()
          @          0x13c8d7d  boost::_bi::bind_t<>::operator()()
          @          0x13c8c90  boost::detail::function::void_function_obj_invoker0<>::invoke()
          @          0x13d68ba  boost::function0<>::operator()()
          @          0x16a1b3d  impala::Thread::SuperviseThread()
          @          0x16aa50c  boost::_bi::list4<>::operator()<>()
          @          0x16aa44f  boost::_bi::bind_t<>::operator()()
          @          0x16aa412  boost::detail::thread_data<>::run()
          @          0x1b8606a  thread_proxy
          @     0x7f8d4de4edf3  start_thread
          @     0x7f8d4db7c1ad  __clone
      I0818 00:43:29.318055 17712 impala-beeswax-server.cc:231] close(): query_id=114d057e65f76f3d:60e4c38f00000000
      I0818 00:43:29.318068 17712 impala-server.cc:991] UnregisterQuery(): query_id=114d057e65f76f3d:60e4c38f00000000
      I0818 00:43:29.318078 17712 impala-server.cc:1077] Cancel(): query_id=114d057e65f76f3d:60e4c38f00000000
      I0818 00:43:29.319111 17712 impala-server.cc:1791] Connection from client ::ffff:127.0.0.1:36947 closed, closing 1 associated session(s)
      I0818 00:43:29.587154 17712 status.cc:122] Session closed
          @          0x12574e3  impala::Status::Status()
          @          0x151835f  impala::ImpalaServer::CloseSessionInternal()
          @          0x151e894  impala::ImpalaServer::ConnectionEnd()
          @          0x13bb2e4  impala::ThriftServer::ThriftServerEventProcessor::deleteContext()
          @          0x29081b3  apache::thrift::server::TThreadPoolServer::Task::run()
          @          0x28efed9  apache::thrift::concurrency::ThreadManager::Worker::run()
          @          0x13c7a6f  impala::ThriftThread::RunRunnable()
          @          0x13c919b  boost::_mfi::mf2<>::operator()()
          @          0x13c9031  boost::_bi::list3<>::operator()<>()
          @          0x13c8d7d  boost::_bi::bind_t<>::operator()()
          @          0x13c8c90  boost::detail::function::void_function_obj_invoker0<>::invoke()
          @          0x13d68ba  boost::function0<>::operator()()
          @          0x16a1b3d  impala::Thread::SuperviseThread()
          @          0x16aa50c  boost::_bi::list4<>::operator()<>()
          @          0x16aa44f  boost::_bi::bind_t<>::operator()()
          @          0x16aa412  boost::detail::thread_data<>::run()
          @          0x1b8606a  thread_proxy
          @     0x7f8d4de4edf3  start_thread
          @     0x7f8d4db7c1ad  __clone
      I0818 00:43:29.596240 17731 impala-server.cc:1391] Catalog topic entry deletion: ROLE:grant_revoke_test_admin
      I0818 00:43:29.596282 17731 impala-server.cc:1391] Catalog topic entry deletion: PRIVILEGE:4.server=server1
      
      Query (id=114d057e65f76f3d:60e4c38f00000000)
        DEBUG MODE WARNING: Query profile created while running a DEBUG build of Impala. Use RELEASE builds to measure query performance.
        Summary
          Session ID: ab4fcedc41970d96:a677076a3a30f086
          Session Type: BEESWAX
          Start Time: 2017-08-18 00:43:29.000313000
          End Time: 2017-08-18 00:43:29.318100000
          Query Type: DDL
          Query State: FINISHED
          Query Status: OK
          Impala Version: impalad version 2.10.0-SNAPSHOT DEBUG (build 68419b2814b5633fa125c456d61cd8f3316f39e9)
          User: jenkins
          Connected User: jenkins
          Delegated User: 
          Network Address: ::ffff:127.0.0.1:36947
          Default Db: default
          Sql Statement: drop role grant_revoke_test_admin
          Coordinator: impala-boost-static-burst-slave-el7-1b82.vpc.cloudera.com:22000
          Query Options (non default): MT_DOP=0
          DDL Type: DROP_ROLE
          
          Query Timeline
            Query submitted: 0
            Planning finished: 1000012
            Request finished: 21000258
            First row fetched: 22000270
            Unregister query: 318003909
        ImpalaServer
          - CatalogOpExecTimer: 15000184
          - ClientFetchWaitTimer: 297003651
          - InactiveTotalTime: 0
          - RowMaterializationTimer: 0
          - TotalTime: 0
      

        Attachments

          Activity

            People

            • Assignee:
              henryr Henry Robinson
              Reporter:
              dknupp David Knupp
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: