Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-4087

TestFragmentLifecycle.test_failure_in_prepare is flaky

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: Impala 2.8.0
    • Fix Version/s: Impala 2.7.0
    • Component/s: Infrastructure
    • Labels:

      Description

      I hit a random failure in TestFragmentLifecycle.test_failure_in_prepare.

      17:26:12 
      17:26:12 =================================== FAILURES ===================================
      17:26:12 ________________ TestFragmentLifecycle.test_failure_in_prepare _________________
      17:26:12 query_test/test_lifecycle.py:49: in test_failure_in_prepare
      17:26:12     v.wait_for_metric_reset(self.IN_FLIGHT_FRAGMENTS)
      17:26:12 verifiers/metric_verifier.py:67: in wait_for_metric_reset
      17:26:12     self.initial_metrics[metric_name], timeout)
      17:26:12 common/impala_service.py:81: in wait_for_metric_value
      17:26:12     (metric_name, expected_value, timeout)
      17:26:12 E   AssertionError: Metric value impala-server.num-fragments-in-flight did not reach value 1 in 60s
      17:26:12 ---------------------------- Captured stderr setup -----------------------------
      17:26:12 -- connecting to: localhost:21000
      17:26:12 ----------------------------- Captured stderr call -----------------------------
      17:26:12 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
      17:26:12 -- executing against localhost:21000
      17:26:12 SET DEBUG_ACTION='-1:0:PREPARE:FAIL';
      17:26:12 
      17:26:12 -- executing against localhost:21000
      17:26:12 SELECT COUNT(*) FROM functional.alltypes;
      17:26:12 
      17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25000
      17:26:12 MainThread: Metric 'impala-server.num-fragments-in-flight' has reach desired value: 0
      17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001
      17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0
      17:26:12 MainThread: Sleeping 1s before next retry.
      17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001
      17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0
      17:26:12 MainThread: Sleeping 1s before next retry.
      17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001
      17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0
      17:26:12 MainThread: Sleeping 1s before next retry.
      17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001
      17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0
      17:26:12 MainThread: Sleeping 1s before next retry.
      17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001
      17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0
      17:26:12 MainThread: Sleeping 1s before next retry.
      17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001
      17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0
      17:26:12 MainThread: Sleeping 1s before next retry.
      17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001
      17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0
      17:26:12 MainThread: Sleeping 1s before next retry.
      17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001
      17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0
      17:26:12 MainThread: Sleeping 1s before next retry.
      17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001
      17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0
      17:26:12 MainThread: Sleeping 1s before next retry.
      17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001
      17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0
      17:26:12 MainThread: Sleeping 1s before next retry.
      17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001
      17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0
      17:26:12 MainThread: Sleeping 1s before next retry.
      17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001
      17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0
      17:26:12 MainThread: Sleeping 1s before next retry.
      17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001
      17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0
      17:26:12 MainThread: Sleeping 1s before next retry.
      17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001
      17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0
      17:26:12 MainThread: Sleeping 1s before next retry.
      17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001
      17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0
      17:26:12 MainThread: Sleeping 1s before next retry.
      17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001
      17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0
      17:26:12 MainThread: Sleeping 1s before next retry.
      17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001
      17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0
      17:26:12 MainThread: Sleeping 1s before next retry.
      17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001
      17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0
      17:26:12 MainThread: Sleeping 1s before next retry.
      17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001
      17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0
      17:26:12 MainThread: Sleeping 1s before next retry.
      17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001
      17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0
      17:26:12 MainThread: Sleeping 1s before next retry.
      17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001
      17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0
      17:26:12 MainThread: Sleeping 1s before next retry.
      17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001
      17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0
      17:26:12 MainThread: Sleeping 1s before next retry.
      17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001
      17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0
      17:26:12 MainThread: Sleeping 1s before next retry.
      17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001
      17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0
      17:26:12 MainThread: Sleeping 1s before next retry.
      17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001
      17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0
      17:26:12 MainThread: Sleeping 1s before next retry.
      17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001
      17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0
      17:26:12 MainThread: Sleeping 1s before next retry.
      17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001
      17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0
      17:26:12 MainThread: Sleeping 1s before next retry.
      17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001
      17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0
      17:26:12 MainThread: Sleeping 1s before next retry.
      17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001
      17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0
      17:26:12 MainThread: Sleeping 1s before next retry.
      17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001
      17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0
      17:26:12 MainThread: Sleeping 1s before next retry.
      17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001
      17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0
      17:26:12 MainThread: Sleeping 1s before next retry.
      17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001
      17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0
      17:26:12 MainThread: Sleeping 1s before next retry.
      17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001
      17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0
      17:26:12 MainThread: Sleeping 1s before next retry.
      17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001
      17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0
      17:26:12 MainThread: Sleeping 1s before next retry.
      17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001
      17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0
      17:26:12 MainThread: Sleeping 1s before next retry.
      17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001
      17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0
      17:26:12 MainThread: Sleeping 1s before next retry.
      17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001
      17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0
      17:26:12 MainThread: Sleeping 1s before next retry.
      17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001
      17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0
      17:26:12 MainThread: Sleeping 1s before next retry.
      17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001
      17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0
      17:26:12 MainThread: Sleeping 1s before next retry.
      17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001
      17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0
      17:26:12 MainThread: Sleeping 1s before next retry.
      17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001
      17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0
      17:26:12 MainThread: Sleeping 1s before next retry.
      17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001
      17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0
      17:26:12 MainThread: Sleeping 1s before next retry.
      17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001
      17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0
      17:26:12 MainThread: Sleeping 1s before next retry.
      17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001
      17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0
      17:26:12 MainThread: Sleeping 1s before next retry.
      17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001
      17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0
      17:26:12 MainThread: Sleeping 1s before next retry.
      17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001
      17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0
      17:26:12 MainThread: Sleeping 1s before next retry.
      17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001
      17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0
      17:26:12 MainThread: Sleeping 1s before next retry.
      17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001
      17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0
      17:26:12 MainThread: Sleeping 1s before next retry.
      17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001
      17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0
      17:26:12 MainThread: Sleeping 1s before next retry.
      17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001
      17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0
      17:26:12 MainThread: Sleeping 1s before next retry.
      17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001
      17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0
      17:26:12 MainThread: Sleeping 1s before next retry.
      17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001
      17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0
      17:26:12 MainThread: Sleeping 1s before next retry.
      17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001
      17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0
      17:26:12 MainThread: Sleeping 1s before next retry.
      17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001
      17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0
      17:26:12 MainThread: Sleeping 1s before next retry.
      17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001
      17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0
      17:26:12 MainThread: Sleeping 1s before next retry.
      17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001
      17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0
      17:26:12 MainThread: Sleeping 1s before next retry.
      17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001
      17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0
      17:26:12 MainThread: Sleeping 1s before next retry.
      17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001
      17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0
      17:26:12 MainThread: Sleeping 1s before next retry.
      17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001
      17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0
      17:26:12 MainThread: Sleeping 1s before next retry.
      17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001
      17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0
      17:26:12 MainThread: Sleeping 1s before next retry.
      17:26:12  generated xml file: /data/jenkins/workspace/impala-external-gerrit-verify-merge-ASF/repos/Impala/logs/ee_tests/results/TEST-impala-serial.xml 
      17:26:12 =========================== short test summary info ============================
      

      It doesn't make sense that it's waiting for the number to go to 1. This is probably because it looks at the current state of the Impala daemon to figure out what the "initial" value is, but we don't make any effort to wait for all fragments to finish before starting the test.

        Attachments

          Activity

            People

            • Assignee:
              tarmstrong Tim Armstrong
              Reporter:
              tarmstrong Tim Armstrong
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: