Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-4642

TestFragmentLifecycle broken; blocking gvos

    Details

      Description

      TestFragmentLifecycle has been failing regularly in runs this weekend, blocking gvos and nightly tests.

      At first I thought this was related to IMPALA-4519 , but now I suspect it's not due to a timing issue and instead something has changed causing the test to fail.

      02:17:51 =================================== FAILURES ===================================
      02:17:51 ________________ TestFragmentLifecycle.test_failure_in_prepare _________________
      02:17:51 query_test/test_lifecycle.py:45: in test_failure_in_prepare
      02:17:51     v.wait_for_metric(self.IN_FLIGHT_FRAGMENTS, 0)
      02:17:51 verifiers/metric_verifier.py:59: in wait_for_metric
      02:17:51     self.impalad_service.wait_for_metric_value(metric_name, expected_value, timeout)
      02:17:51 common/impala_service.py:85: in wait_for_metric_value
      02:17:51     (metric_name, expected_value, timeout)
      02:17:51 E   AssertionError: Metric value impala-server.num-fragments-in-flight did not reach value 0 in 60s
      02:17:51 ---------------------------- Captured stderr setup -----------------------------
      02:17:51 -- connecting to: localhost:21000
      02:17:51 ----------------------------- Captured stderr call -----------------------------
      02:17:51 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
      02:17:51 -- executing against localhost:21000
      02:17:51 SET DEBUG_ACTION='-1:0:PREPARE:FAIL';
      02:17:51 
      02:17:51 -- executing against localhost:21000
      02:17:51 SELECT COUNT(*) FROM functional.alltypes;
      02:17:51 
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000
      02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2
      02:17:51 MainThread: Sleeping 1s before next retry.
      02:17:51  generated xml file: /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/logs/ee_tests/results/TEST-impala-serial.xml 
      02:17:51 =========================== short test summary info ============================
      

        Attachments

          Activity

            People

            • Assignee:
              mjacobs Matthew Jacobs
              Reporter:
              mjacobs Matthew Jacobs
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: