Details
-
Bug
-
Status: Resolved
-
Blocker
-
Resolution: Fixed
-
Impala 2.8.0
Description
TestFragmentLifecycle has been failing regularly in runs this weekend, blocking gvos and nightly tests.
At first I thought this was related to IMPALA-4519 , but now I suspect it's not due to a timing issue and instead something has changed causing the test to fail.
02:17:51 =================================== FAILURES =================================== 02:17:51 ________________ TestFragmentLifecycle.test_failure_in_prepare _________________ 02:17:51 query_test/test_lifecycle.py:45: in test_failure_in_prepare 02:17:51 v.wait_for_metric(self.IN_FLIGHT_FRAGMENTS, 0) 02:17:51 verifiers/metric_verifier.py:59: in wait_for_metric 02:17:51 self.impalad_service.wait_for_metric_value(metric_name, expected_value, timeout) 02:17:51 common/impala_service.py:85: in wait_for_metric_value 02:17:51 (metric_name, expected_value, timeout) 02:17:51 E AssertionError: Metric value impala-server.num-fragments-in-flight did not reach value 0 in 60s 02:17:51 ---------------------------- Captured stderr setup ----------------------------- 02:17:51 -- connecting to: localhost:21000 02:17:51 ----------------------------- Captured stderr call ----------------------------- 02:17:51 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es) 02:17:51 -- executing against localhost:21000 02:17:51 SET DEBUG_ACTION='-1:0:PREPARE:FAIL'; 02:17:51 02:17:51 -- executing against localhost:21000 02:17:51 SELECT COUNT(*) FROM functional.alltypes; 02:17:51 02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000 02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2 02:17:51 MainThread: Sleeping 1s before next retry. 02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000 02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2 02:17:51 MainThread: Sleeping 1s before next retry. 02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000 02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2 02:17:51 MainThread: Sleeping 1s before next retry. 02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000 02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2 02:17:51 MainThread: Sleeping 1s before next retry. 02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000 02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2 02:17:51 MainThread: Sleeping 1s before next retry. 02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000 02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2 02:17:51 MainThread: Sleeping 1s before next retry. 02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000 02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2 02:17:51 MainThread: Sleeping 1s before next retry. 02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000 02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2 02:17:51 MainThread: Sleeping 1s before next retry. 02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000 02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2 02:17:51 MainThread: Sleeping 1s before next retry. 02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000 02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2 02:17:51 MainThread: Sleeping 1s before next retry. 02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000 02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2 02:17:51 MainThread: Sleeping 1s before next retry. 02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000 02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2 02:17:51 MainThread: Sleeping 1s before next retry. 02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000 02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2 02:17:51 MainThread: Sleeping 1s before next retry. 02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000 02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2 02:17:51 MainThread: Sleeping 1s before next retry. 02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000 02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2 02:17:51 MainThread: Sleeping 1s before next retry. 02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000 02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2 02:17:51 MainThread: Sleeping 1s before next retry. 02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000 02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2 02:17:51 MainThread: Sleeping 1s before next retry. 02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000 02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2 02:17:51 MainThread: Sleeping 1s before next retry. 02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000 02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2 02:17:51 MainThread: Sleeping 1s before next retry. 02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000 02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2 02:17:51 MainThread: Sleeping 1s before next retry. 02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000 02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2 02:17:51 MainThread: Sleeping 1s before next retry. 02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000 02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2 02:17:51 MainThread: Sleeping 1s before next retry. 02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000 02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2 02:17:51 MainThread: Sleeping 1s before next retry. 02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000 02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2 02:17:51 MainThread: Sleeping 1s before next retry. 02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000 02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2 02:17:51 MainThread: Sleeping 1s before next retry. 02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000 02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2 02:17:51 MainThread: Sleeping 1s before next retry. 02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000 02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2 02:17:51 MainThread: Sleeping 1s before next retry. 02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000 02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2 02:17:51 MainThread: Sleeping 1s before next retry. 02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000 02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2 02:17:51 MainThread: Sleeping 1s before next retry. 02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000 02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2 02:17:51 MainThread: Sleeping 1s before next retry. 02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000 02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2 02:17:51 MainThread: Sleeping 1s before next retry. 02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000 02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2 02:17:51 MainThread: Sleeping 1s before next retry. 02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000 02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2 02:17:51 MainThread: Sleeping 1s before next retry. 02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000 02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2 02:17:51 MainThread: Sleeping 1s before next retry. 02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000 02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2 02:17:51 MainThread: Sleeping 1s before next retry. 02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000 02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2 02:17:51 MainThread: Sleeping 1s before next retry. 02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000 02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2 02:17:51 MainThread: Sleeping 1s before next retry. 02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000 02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2 02:17:51 MainThread: Sleeping 1s before next retry. 02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000 02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2 02:17:51 MainThread: Sleeping 1s before next retry. 02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000 02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2 02:17:51 MainThread: Sleeping 1s before next retry. 02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000 02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2 02:17:51 MainThread: Sleeping 1s before next retry. 02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000 02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2 02:17:51 MainThread: Sleeping 1s before next retry. 02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000 02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2 02:17:51 MainThread: Sleeping 1s before next retry. 02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000 02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2 02:17:51 MainThread: Sleeping 1s before next retry. 02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000 02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2 02:17:51 MainThread: Sleeping 1s before next retry. 02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000 02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2 02:17:51 MainThread: Sleeping 1s before next retry. 02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000 02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2 02:17:51 MainThread: Sleeping 1s before next retry. 02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000 02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2 02:17:51 MainThread: Sleeping 1s before next retry. 02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000 02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2 02:17:51 MainThread: Sleeping 1s before next retry. 02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000 02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2 02:17:51 MainThread: Sleeping 1s before next retry. 02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000 02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2 02:17:51 MainThread: Sleeping 1s before next retry. 02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000 02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2 02:17:51 MainThread: Sleeping 1s before next retry. 02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000 02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2 02:17:51 MainThread: Sleeping 1s before next retry. 02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000 02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2 02:17:51 MainThread: Sleeping 1s before next retry. 02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000 02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2 02:17:51 MainThread: Sleeping 1s before next retry. 02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000 02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2 02:17:51 MainThread: Sleeping 1s before next retry. 02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000 02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2 02:17:51 MainThread: Sleeping 1s before next retry. 02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000 02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2 02:17:51 MainThread: Sleeping 1s before next retry. 02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000 02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2 02:17:51 MainThread: Sleeping 1s before next retry. 02:17:51 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-0f96.vpc.cloudera.com:25000 02:17:51 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=0. Current value: 2 02:17:51 MainThread: Sleeping 1s before next retry. 02:17:51 generated xml file: /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/logs/ee_tests/results/TEST-impala-serial.xml 02:17:51 =========================== short test summary info ============================