[IMPALA-4519] TestFragmentLifecycle is flaky

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 2.8.0
• Fix Version/s: Impala 2.8.0
• Component/s: Infrastructure
• Labels: broken-build flaky
• Target Version: Impala 2.8.0
• Assignee: Tim Armstrong
• Reporter: Thomas Tauber-Marshall
• Created: 22/Nov/16 20:53
• Updated: 24/Jan/17 04:16
• Resolved: 11/Dec/16 16:16

Description

Saw this failure on a Jenkins job for both test_failure_in_prepare and test_failure_in_prepare_multi_fragment.

Tim - assigning this to you as it may be related to IMPALA-4087

11:09:56 =================================== FAILURES ===================================
11:09:56 ________________ TestFragmentLifecycle.test_failure_in_prepare _________________
11:09:56 query_test/test_lifecycle.py:45: in test_failure_in_prepare
11:09:56     v.wait_for_metric(self.IN_FLIGHT_FRAGMENTS, 0)
11:09:56 verifiers/metric_verifier.py:59: in wait_for_metric
11:09:56     self.impalad_service.wait_for_metric_value(metric_name, expected_value, timeout)
11:09:56 common/impala_service.py:85: in wait_for_metric_value
11:09:56     (metric_name, expected_value, timeout)
11:09:56 E   AssertionError: Metric value impala-server.num-fragments-in-flight did not reach value 0 in 60s
11:09:56 ---------------------------- Captured stderr setup -----------------------------
11:09:56 -- connecting to: localhost:21000
11:09:56 ----------------------------- Captured stderr call -----------------------------
11:09:56 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
11:09:56 -- executing against localhost:21000
11:09:56 SET DEBUG_ACTION='-1:0:PREPARE:FAIL';
11:09:56 
11:09:56 -- executing against localhost:21000
11:09:56 SELECT COUNT(*) FROM functional.alltypes;
11:09:56 
11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 3
11:09:56 MainThread: Sleeping 1s before next retry.
11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 3
11:09:56 MainThread: Sleeping 1s before next retry.
11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
11:09:56 MainThread: Sleeping 1s before next retry.
11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
11:09:56 MainThread: Sleeping 1s before next retry.
11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
11:09:56 MainThread: Sleeping 1s before next retry.
11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
11:09:56 MainThread: Sleeping 1s before next retry.
11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
11:09:56 MainThread: Sleeping 1s before next retry.
11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
11:09:56 MainThread: Sleeping 1s before next retry.
11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
11:09:56 MainThread: Sleeping 1s before next retry.
11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
11:09:56 MainThread: Sleeping 1s before next retry.
11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
11:09:56 MainThread: Sleeping 1s before next retry.
11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
11:09:56 MainThread: Sleeping 1s before next retry.
11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
11:09:56 MainThread: Sleeping 1s before next retry.
11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
11:09:56 MainThread: Sleeping 1s before next retry.
11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
11:09:56 MainThread: Sleeping 1s before next retry.
11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
11:09:56 MainThread: Sleeping 1s before next retry.
11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
11:09:56 MainThread: Sleeping 1s before next retry.
11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
11:09:56 MainThread: Sleeping 1s before next retry.
11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
11:09:56 MainThread: Sleeping 1s before next retry.
11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
11:09:56 MainThread: Sleeping 1s before next retry.
11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
11:09:56 MainThread: Sleeping 1s before next retry.
11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
11:09:56 MainThread: Sleeping 1s before next retry.
11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
11:09:56 MainThread: Sleeping 1s before next retry.
11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
11:09:56 MainThread: Sleeping 1s before next retry.
11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
11:09:56 MainThread: Sleeping 1s before next retry.
11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
11:09:56 MainThread: Sleeping 1s before next retry.
11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
11:09:56 MainThread: Sleeping 1s before next retry.
11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
11:09:56 MainThread: Sleeping 1s before next retry.
11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
11:09:56 MainThread: Sleeping 1s before next retry.
11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
11:09:56 MainThread: Sleeping 1s before next retry.
11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
11:09:56 MainThread: Sleeping 1s before next retry.
11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
11:09:56 MainThread: Sleeping 1s before next retry.
11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
11:09:56 MainThread: Sleeping 1s before next retry.
11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
11:09:56 MainThread: Sleeping 1s before next retry.
11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
11:09:56 MainThread: Sleeping 1s before next retry.
11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
11:09:56 MainThread: Sleeping 1s before next retry.
11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
11:09:56 MainThread: Sleeping 1s before next retry.
11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
11:09:56 MainThread: Sleeping 1s before next retry.
11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
11:09:56 MainThread: Sleeping 1s before next retry.
11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
11:09:56 MainThread: Sleeping 1s before next retry.
11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
11:09:56 MainThread: Sleeping 1s before next retry.
11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
11:09:56 MainThread: Sleeping 1s before next retry.
11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
11:09:56 MainThread: Sleeping 1s before next retry.
11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
11:09:56 MainThread: Sleeping 1s before next retry.
11:09:56 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-14d9.vpc.cloudera.com:25000
11:09:56 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 1
...