Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-3912

test_random_rpc_timeout is flaky

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: Impala 2.7.0
    • Fix Version/s: None
    • Component/s: Distributed Exec
    • Labels:

      Description

      =================================== FAILURES ===================================
       TestRPCTimeout.test_random_rpc_timeout[exec_option: {'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0, 'batch_size': 0, 'num_nodes': 0} | table_format: text/none] 
      
      self = <test_rpc_timeout.TestRPCTimeout object at 0x5c6ff90>
      vector = <tests.common.test_vector.TestVector object at 0x5f9cbd0>
      
          @pytest.mark.execute_serially
          @CustomClusterTestSuite.with_args("--backend_client_rpc_timeout_ms=1000"
              " --fault_injection_rpc_delay_ms=3000 --fault_injection_rpc_type=7")
          def test_random_rpc_timeout(self, vector):
      >     self.execute_query_verify_metrics(self.TEST_QUERY, 10)
      
      custom_cluster/test_rpc_timeout.py:125: 
      _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
      custom_cluster/test_rpc_timeout.py:48: in execute_query_verify_metrics
          v.wait_for_metric("impala-server.num-fragments-in-flight", 0)
      verifiers/metric_verifier.py:59: in wait_for_metric
          self.impalad_service.wait_for_metric_value(metric_name, expected_value, timeout)
      _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
      
      self = <tests.common.impala_service.ImpaladService object at 0x5047050>
      metric_name = 'impala-server.num-fragments-in-flight', expected_value = 0
      timeout = 60, interval = 1
      
          def wait_for_metric_value(self, metric_name, expected_value, timeout=10, interval=1):
            start_time = time()
            while (time() - start_time < timeout):
              LOG.info("Getting metric: %s from %s:%s" %
                  (metric_name, self.hostname, self.webserver_port))
              value = None
              try:
                value = self.get_metric_value(metric_name)
              except Exception, e:
                LOG.error(e)
          
              if value == expected_value:
                LOG.info("Metric '%s' has reach desired value: %s" % (metric_name, value))
                return value
              else:
                LOG.info("Waiting for metric value '%s'=%s. Current value: %s" %
                    (metric_name, expected_value, value))
              LOG.info("Sleeping %ds before next retry." % interval)
              sleep(interval)
            assert 0, 'Metric value %s did not reach value %s in %ss' %\
      >         (metric_name, expected_value, timeout)
      E     AssertionError: Metric value impala-server.num-fragments-in-flight did not reach value 0 in 60s
      
      common/impala_service.py:78: AssertionError
      

      http://sandbox.jenkins.cloudera.com/job/impala-asf-master-exhaustive-rhel7/22/

        Attachments

          Activity

            People

            • Assignee:
              jyu@cloudera.com Juan Yu
              Reporter:
              jbapple Jim Apple
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: