Details

      Description

      Saw this failure on a Jenkins job for both test_failure_in_prepare and test_failure_in_prepare_multi_fragment.

      Tim - assigning this to you as it may be related to IMPALA-4087

      11:09:56 =================================== FAILURES ===================================
      11:09:56 ________________ TestFragmentLifecycle.test_failure_in_prepare _________________
      11:09:56 query_test/test_lifecycle.py:45: in test_failure_in_prepare
      11:09:56     v.wait_for_metric(self.IN_FLIGHT_FRAGMENTS, 0)
      11:09:56 verifiers/metric_verifier.py:59: in wait_for_metric
      11:09:56     self.impalad_service.wait_for_metric_value(metric_name, expected_value, timeout)
      11:09:56 common/impala_service.py:85: in wait_for_metric_value
      11:09:56     (metric_name, expected_value, timeout)
      11:09:56 E   AssertionError: Metric value impala-server.num-fragments-in-flight did not reach value 0 in 60s
      11:09:56 ---------------------------- Captured stderr setup -----------------------------
      11:09:56 -- connecting to: localhost:21000
      11:09:56 ----------------------------- Captured stderr call -----------------------------
      11:09:56 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
      11:09:56 -- executing against localhost:21000
      11:09:56 SET DEBUG_ACTION='-1:0:PREPARE:FAIL';
      11:09:56 
      11:09:56 -- executing against localhost:21000
      11:09:56 SELECT COUNT(*) FROM functional.alltypes;
      11:09:56 
      11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
      11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 3
      11:09:56 MainThread: Sleeping 1s before next retry.
      11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
      11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 3
      11:09:56 MainThread: Sleeping 1s before next retry.
      11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
      11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
      11:09:56 MainThread: Sleeping 1s before next retry.
      11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
      11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
      11:09:56 MainThread: Sleeping 1s before next retry.
      11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
      11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
      11:09:56 MainThread: Sleeping 1s before next retry.
      11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
      11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
      11:09:56 MainThread: Sleeping 1s before next retry.
      11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
      11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
      11:09:56 MainThread: Sleeping 1s before next retry.
      11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
      11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
      11:09:56 MainThread: Sleeping 1s before next retry.
      11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
      11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
      11:09:56 MainThread: Sleeping 1s before next retry.
      11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
      11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
      11:09:56 MainThread: Sleeping 1s before next retry.
      11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
      11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
      11:09:56 MainThread: Sleeping 1s before next retry.
      11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
      11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
      11:09:56 MainThread: Sleeping 1s before next retry.
      11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
      11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
      11:09:56 MainThread: Sleeping 1s before next retry.
      11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
      11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
      11:09:56 MainThread: Sleeping 1s before next retry.
      11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
      11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
      11:09:56 MainThread: Sleeping 1s before next retry.
      11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
      11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
      11:09:56 MainThread: Sleeping 1s before next retry.
      11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
      11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
      11:09:56 MainThread: Sleeping 1s before next retry.
      11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
      11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
      11:09:56 MainThread: Sleeping 1s before next retry.
      11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
      11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
      11:09:56 MainThread: Sleeping 1s before next retry.
      11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
      11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
      11:09:56 MainThread: Sleeping 1s before next retry.
      11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
      11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
      11:09:56 MainThread: Sleeping 1s before next retry.
      11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
      11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
      11:09:56 MainThread: Sleeping 1s before next retry.
      11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
      11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
      11:09:56 MainThread: Sleeping 1s before next retry.
      11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
      11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
      11:09:56 MainThread: Sleeping 1s before next retry.
      11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
      11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
      11:09:56 MainThread: Sleeping 1s before next retry.
      11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
      11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
      11:09:56 MainThread: Sleeping 1s before next retry.
      11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
      11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
      11:09:56 MainThread: Sleeping 1s before next retry.
      11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
      11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
      11:09:56 MainThread: Sleeping 1s before next retry.
      11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
      11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
      11:09:56 MainThread: Sleeping 1s before next retry.
      11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
      11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
      11:09:56 MainThread: Sleeping 1s before next retry.
      11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
      11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
      11:09:56 MainThread: Sleeping 1s before next retry.
      11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
      11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
      11:09:56 MainThread: Sleeping 1s before next retry.
      11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
      11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
      11:09:56 MainThread: Sleeping 1s before next retry.
      11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
      11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
      11:09:56 MainThread: Sleeping 1s before next retry.
      11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
      11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
      11:09:56 MainThread: Sleeping 1s before next retry.
      11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
      11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
      11:09:56 MainThread: Sleeping 1s before next retry.
      11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
      11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
      11:09:56 MainThread: Sleeping 1s before next retry.
      11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
      11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
      11:09:56 MainThread: Sleeping 1s before next retry.
      11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
      11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
      11:09:56 MainThread: Sleeping 1s before next retry.
      11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
      11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
      11:09:56 MainThread: Sleeping 1s before next retry.
      11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
      11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
      11:09:56 MainThread: Sleeping 1s before next retry.
      11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
      11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
      11:09:56 MainThread: Sleeping 1s before next retry.
      11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
      11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
      11:09:56 MainThread: Sleeping 1s before next retry.
      11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
      11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
      11:09:56 MainThread: Sleeping 1s before next retry.
      11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
      11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
      ...

        Attachments

          Activity

            People

            • Assignee:
              tarmstrong Tim Armstrong
              Reporter:
              twmarshall Thomas Tauber-Marshall
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: