Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-4238

custom_cluster/test_client_ssl.py TestClientSsl.test_ssl AssertionError: SIGINT was not caught by shell within 30s

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: Impala 2.8.0, Impala 2.10.0
    • Fix Version/s: Impala 4.0.0
    • Component/s: Security

      Description

      asf master core test failure: http://sandbox.jenkins.sf.cloudera.com/view/Impala/view/Evergreen-asf-master/job/impala-asf-master-core/540/

      http://sandbox.jenkins.sf.cloudera.com/job/impala-umbrella-build-and-test/4921/console

      08:18:51 =================================== FAILURES ===================================
      08:18:51  TestClientSsl.test_ssl[exec_option: {'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0, 'batch_size': 0, 'num_nodes': 0} | table_format: text/none] 
      08:18:51 
      08:18:51 self = <test_client_ssl.TestClientSsl object at 0x4b40810>
      08:18:51 vector = <tests.common.test_vector.TestVector object at 0x4ac9110>
      08:18:51 
      08:18:51     @pytest.mark.execute_serially
      08:18:51     @CustomClusterTestSuite.with_args("--ssl_server_certificate=%s/server-cert.pem "
      08:18:51                                       "--ssl_private_key=%s/server-key.pem"
      08:18:51                                       % (CERT_DIR, CERT_DIR))
      08:18:51     def test_ssl(self, vector):
      08:18:51     
      08:18:51       self._verify_negative_cases()
      08:18:51       # TODO: This is really two different tests, but the custom cluster takes too long to
      08:18:51       # start. Make it so that custom clusters can be specified across test suites.
      08:18:51       self._validate_positive_cases("%s/server-cert.pem" % self.CERT_DIR)
      08:18:51     
      08:18:51       # No certificate checking: will accept any cert.
      08:18:51       self._validate_positive_cases()
      08:18:51     
      08:18:51       # Test cancelling a query
      08:18:51       impalad = ImpaladService(socket.getfqdn())
      08:18:51       impalad.wait_for_num_in_flight_queries(0)
      08:18:51       p = ImpalaShell(args="--ssl")
      08:18:51       p.send_cmd("SET DEBUG_ACTION=0:OPEN:WAIT")
      08:18:51       p.send_cmd("select count(*) from functional.alltypes")
      08:18:51       impalad.wait_for_num_in_flight_queries(1)
      08:18:51     
      08:18:51       LOG = logging.getLogger('test_client_ssl')
      08:18:51       LOG.info("Cancelling query")
      08:18:51       num_tries = 0
      08:18:51       # In practice, sending SIGINT to the shell process doesn't always seem to get caught
      08:18:51       # (and a search shows up some bugs in Python where SIGINT might be ignored). So retry
      08:18:51       # for 30s until one signal takes.
      08:18:51       while impalad.get_num_in_flight_queries() == 1:
      08:18:51         time.sleep(1)
      08:18:51         LOG.info("Sending signal...")
      08:18:51         os.kill(p.pid(), signal.SIGINT)
      08:18:51         num_tries += 1
      08:18:51 >       assert num_tries < 30, "SIGINT was not caught by shell within 30s"
      08:18:51 E       AssertionError: SIGINT was not caught by shell within 30s
      08:18:51 E       assert 30 < 30
      08:18:51 
      08:18:51 custom_cluster/test_client_ssl.py:85: AssertionError
      08:18:51 ---------------------------- Captured stdout setup -----------------------------
      08:18:51 Starting State Store logging to /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/logs/custom_cluster_tests/statestored.INFO
      08:18:51 Starting Catalog Service logging to /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/logs/custom_cluster_tests/catalogd.INFO
      08:18:51 Starting Impala Daemon logging to /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/logs/custom_cluster_tests/impalad.INFO
      08:18:51 Starting Impala Daemon logging to /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/logs/custom_cluster_tests/impalad_node1.INFO
      08:18:51 Starting Impala Daemon logging to /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/logs/custom_cluster_tests/impalad_node2.INFO
      08:18:51 Waiting for Catalog... Status: 53 DBs / 1077 tables (ready=True)
      08:18:51 Waiting for Catalog... Status: 53 DBs / 1077 tables (ready=True)
      08:18:51 Waiting for Catalog... Status: 53 DBs / 1077 tables (ready=True)
      08:18:51 Impala Cluster Running with 3 nodes.
      08:18:51 ---------------------------- Captured stderr setup -----------------------------
      08:18:51 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
      08:18:51 MainThread: Getting num_known_live_backends from impala-boost-static-burst-slave-07f9.vpc.cloudera.com:25000
      08:18:51 MainThread: Debug webpage not yet available.
      08:18:51 MainThread: Waiting for num_known_live_backends=3. Current value: 0
      08:18:51 MainThread: Getting num_known_live_backends from impala-boost-static-burst-slave-07f9.vpc.cloudera.com:25000
      08:18:51 MainThread: Waiting for num_known_live_backends=3. Current value: 2
      08:18:51 MainThread: Getting num_known_live_backends from impala-boost-static-burst-slave-07f9.vpc.cloudera.com:25000
      08:18:51 MainThread: Waiting for num_known_live_backends=3. Current value: 2
      08:18:51 MainThread: Getting num_known_live_backends from impala-boost-static-burst-slave-07f9.vpc.cloudera.com:25000
      08:18:51 MainThread: num_known_live_backends has reached value: 3
      08:18:51 MainThread: Getting num_known_live_backends from impala-boost-static-burst-slave-07f9.vpc.cloudera.com:25001
      08:18:51 MainThread: num_known_live_backends has reached value: 3
      08:18:51 MainThread: Getting num_known_live_backends from impala-boost-static-burst-slave-07f9.vpc.cloudera.com:25002
      08:18:51 MainThread: num_known_live_backends has reached value: 3
      08:18:51 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
      08:18:51 MainThread: Getting metric: statestore.live-backends from impala-boost-static-burst-slave-07f9.vpc.cloudera.com:25010
      08:18:51 MainThread: Metric 'statestore.live-backends' has reach desired value: 4
      08:18:51 MainThread: Getting num_known_live_backends from impala-boost-static-burst-slave-07f9.vpc.cloudera.com:25000
      08:18:51 MainThread: num_known_live_backends has reached value: 3
      08:18:51 MainThread: Getting num_known_live_backends from impala-boost-static-burst-slave-07f9.vpc.cloudera.com:25001
      08:18:51 MainThread: num_known_live_backends has reached value: 3
      08:18:51 MainThread: Getting num_known_live_backends from impala-boost-static-burst-slave-07f9.vpc.cloudera.com:25002
      08:18:51 MainThread: num_known_live_backends has reached value: 3
      08:18:51 -- connecting to: localhost:21000
      08:18:51 ----------------------------- Captured stderr call -----------------------------
      08:18:51 MainThread: Getting num_in_flight_queries from localhost.localdomain:25000
      08:18:51 MainThread: Getting num_in_flight_queries from localhost.localdomain:25000
      08:18:51 MainThread: Getting num_in_flight_queries from localhost.localdomain:25000
      08:18:51 MainThread: Cancelling query
      08:18:51 MainThread: Getting num_in_flight_queries from localhost.localdomain:25000
      08:18:51 MainThread: Sending signal...
      08:18:51 MainThread: Getting num_in_flight_queries from localhost.localdomain:25000
      08:18:51 MainThread: Sending signal...
      08:18:51 MainThread: Getting num_in_flight_queries from localhost.localdomain:25000
      08:18:51 MainThread: Sending signal...
      08:18:51 MainThread: Getting num_in_flight_queries from localhost.localdomain:25000
      08:18:51 MainThread: Sending signal...
      08:18:51 MainThread: Getting num_in_flight_queries from localhost.localdomain:25000
      08:18:51 MainThread: Sending signal...
      08:18:51 MainThread: Getting num_in_flight_queries from localhost.localdomain:25000
      08:18:51 MainThread: Sending signal...
      08:18:51 MainThread: Getting num_in_flight_queries from localhost.localdomain:25000
      08:18:51 MainThread: Sending signal...
      08:18:51 MainThread: Getting num_in_flight_queries from localhost.localdomain:25000
      08:18:51 MainThread: Sending signal...
      08:18:51 MainThread: Getting num_in_flight_queries from localhost.localdomain:25000
      08:18:51 MainThread: Sending signal...
      08:18:51 MainThread: Getting num_in_flight_queries from localhost.localdomain:25000
      08:18:51 MainThread: Sending signal...
      08:18:51 MainThread: Getting num_in_flight_queries from localhost.localdomain:25000
      08:18:51 MainThread: Sending signal...
      08:18:51 MainThread: Getting num_in_flight_queries from localhost.localdomain:25000
      08:18:51 MainThread: Sending signal...
      08:18:51 MainThread: Getting num_in_flight_queries from localhost.localdomain:25000
      08:18:51 MainThread: Sending signal...
      08:18:51 MainThread: Getting num_in_flight_queries from localhost.localdomain:25000
      08:18:51 MainThread: Sending signal...
      08:18:51 MainThread: Getting num_in_flight_queries from localhost.localdomain:25000
      08:18:51 MainThread: Sending signal...
      08:18:51 MainThread: Getting num_in_flight_queries from localhost.localdomain:25000
      08:18:51 MainThread: Sending signal...
      08:18:51 MainThread: Getting num_in_flight_queries from localhost.localdomain:25000
      08:18:51 MainThread: Sending signal...
      08:18:51 MainThread: Getting num_in_flight_queries from localhost.localdomain:25000
      08:18:51 MainThread: Sending signal...
      08:18:51 MainThread: Getting num_in_flight_queries from localhost.localdomain:25000
      08:18:51 MainThread: Sending signal...
      08:18:51 MainThread: Getting num_in_flight_queries from localhost.localdomain:25000
      08:18:51 MainThread: Sending signal...
      08:18:51 MainThread: Getting num_in_flight_queries from localhost.localdomain:25000
      08:18:51 MainThread: Sending signal...
      08:18:51 MainThread: Getting num_in_flight_queries from localhost.localdomain:25000
      08:18:51 MainThread: Sending signal...
      08:18:51 MainThread: Getting num_in_flight_queries from localhost.localdomain:25000
      08:18:51 MainThread: Sending signal...
      08:18:51 MainThread: Getting num_in_flight_queries from localhost.localdomain:25000
      08:18:51 MainThread: Sending signal...
      08:18:51 MainThread: Getting num_in_flight_queries from localhost.localdomain:25000
      08:18:51 MainThread: Sending signal...
      08:18:51 MainThread: Getting num_in_flight_queries from localhost.localdomain:25000
      08:18:51 MainThread: Sending signal...
      08:18:51 MainThread: Getting num_in_flight_queries from localhost.localdomain:25000
      08:18:51 MainThread: Sending signal...
      08:18:51 MainThread: Getting num_in_flight_queries from localhost.localdomain:25000
      08:18:51 MainThread: Sending signal...
      08:18:51 MainThread: Getting num_in_flight_queries from localhost.localdomain:25000
      08:18:51 MainThread: Sending signal...
      08:18:51 MainThread: Getting num_in_flight_queries from localhost.localdomain:25000
      08:18:51 MainThread: Sending signal...
      08:18:51  generated xml file: /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/logs/custom_cluster_tests/results/TEST-impala-custom-cluster.xml 
      08:18:51 == 1 failed, 42 passed, 20 skipped, 6 xfailed, 2 warnings in 3094.24 seconds ===
      08:18:51 Error in /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/tests/run-custom-cluster-tests.sh at line 42: impala-py.test custom_cluster/ authorization/ ${AUX_CUSTOM_DIR} \
      08:18:51 Error in /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/buildall.sh at line 412: "${IMPALA_HOME}/bin/run-all-tests.sh" -e $EXPLORATION_STRATEGY
      08:18:51 + echo 'buildall.sh ' -format '-snapshot_file /data/jenkins/workspace/impala-umbrella-build-and-test/testdata/test-warehouse-SNAPSHOT/test-warehouse-cdh5-85-SNAPSHOT.tar.gz' '-metastore_snapshot_file /data/jenkins/workspace/impala-umbrella-build-and-test/testdata/hive_metastore_snapshot/hive_impala_dump_cdh5-85.txt failed.'
      08:18:51 buildall.sh  -format -snapshot_file /data/jenkins/workspace/impala-umbrella-build-and-test/testdata/test-warehouse-SNAPSHOT/test-warehouse-cdh5-85-SNAPSHOT.tar.gz -metastore_snapshot_file /data/jenkins/workspace/impala-umbrella-build-and-test/testdata/hive_metastore_snapshot/hive_impala_dump_cdh5-85.txt failed.
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                tarmstrong Tim Armstrong
                Reporter:
                hsheinblatt_impala_e511 Harrison Sheinblatt
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: