Details
-
Bug
-
Status: Resolved
-
Critical
-
Resolution: Fixed
-
Impala 2.8.0
Description
I hit a random failure in TestFragmentLifecycle.test_failure_in_prepare.
17:26:12 17:26:12 =================================== FAILURES =================================== 17:26:12 ________________ TestFragmentLifecycle.test_failure_in_prepare _________________ 17:26:12 query_test/test_lifecycle.py:49: in test_failure_in_prepare 17:26:12 v.wait_for_metric_reset(self.IN_FLIGHT_FRAGMENTS) 17:26:12 verifiers/metric_verifier.py:67: in wait_for_metric_reset 17:26:12 self.initial_metrics[metric_name], timeout) 17:26:12 common/impala_service.py:81: in wait_for_metric_value 17:26:12 (metric_name, expected_value, timeout) 17:26:12 E AssertionError: Metric value impala-server.num-fragments-in-flight did not reach value 1 in 60s 17:26:12 ---------------------------- Captured stderr setup ----------------------------- 17:26:12 -- connecting to: localhost:21000 17:26:12 ----------------------------- Captured stderr call ----------------------------- 17:26:12 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es) 17:26:12 -- executing against localhost:21000 17:26:12 SET DEBUG_ACTION='-1:0:PREPARE:FAIL'; 17:26:12 17:26:12 -- executing against localhost:21000 17:26:12 SELECT COUNT(*) FROM functional.alltypes; 17:26:12 17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25000 17:26:12 MainThread: Metric 'impala-server.num-fragments-in-flight' has reach desired value: 0 17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001 17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0 17:26:12 MainThread: Sleeping 1s before next retry. 17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001 17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0 17:26:12 MainThread: Sleeping 1s before next retry. 17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001 17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0 17:26:12 MainThread: Sleeping 1s before next retry. 17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001 17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0 17:26:12 MainThread: Sleeping 1s before next retry. 17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001 17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0 17:26:12 MainThread: Sleeping 1s before next retry. 17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001 17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0 17:26:12 MainThread: Sleeping 1s before next retry. 17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001 17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0 17:26:12 MainThread: Sleeping 1s before next retry. 17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001 17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0 17:26:12 MainThread: Sleeping 1s before next retry. 17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001 17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0 17:26:12 MainThread: Sleeping 1s before next retry. 17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001 17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0 17:26:12 MainThread: Sleeping 1s before next retry. 17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001 17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0 17:26:12 MainThread: Sleeping 1s before next retry. 17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001 17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0 17:26:12 MainThread: Sleeping 1s before next retry. 17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001 17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0 17:26:12 MainThread: Sleeping 1s before next retry. 17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001 17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0 17:26:12 MainThread: Sleeping 1s before next retry. 17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001 17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0 17:26:12 MainThread: Sleeping 1s before next retry. 17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001 17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0 17:26:12 MainThread: Sleeping 1s before next retry. 17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001 17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0 17:26:12 MainThread: Sleeping 1s before next retry. 17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001 17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0 17:26:12 MainThread: Sleeping 1s before next retry. 17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001 17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0 17:26:12 MainThread: Sleeping 1s before next retry. 17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001 17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0 17:26:12 MainThread: Sleeping 1s before next retry. 17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001 17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0 17:26:12 MainThread: Sleeping 1s before next retry. 17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001 17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0 17:26:12 MainThread: Sleeping 1s before next retry. 17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001 17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0 17:26:12 MainThread: Sleeping 1s before next retry. 17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001 17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0 17:26:12 MainThread: Sleeping 1s before next retry. 17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001 17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0 17:26:12 MainThread: Sleeping 1s before next retry. 17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001 17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0 17:26:12 MainThread: Sleeping 1s before next retry. 17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001 17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0 17:26:12 MainThread: Sleeping 1s before next retry. 17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001 17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0 17:26:12 MainThread: Sleeping 1s before next retry. 17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001 17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0 17:26:12 MainThread: Sleeping 1s before next retry. 17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001 17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0 17:26:12 MainThread: Sleeping 1s before next retry. 17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001 17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0 17:26:12 MainThread: Sleeping 1s before next retry. 17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001 17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0 17:26:12 MainThread: Sleeping 1s before next retry. 17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001 17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0 17:26:12 MainThread: Sleeping 1s before next retry. 17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001 17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0 17:26:12 MainThread: Sleeping 1s before next retry. 17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001 17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0 17:26:12 MainThread: Sleeping 1s before next retry. 17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001 17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0 17:26:12 MainThread: Sleeping 1s before next retry. 17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001 17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0 17:26:12 MainThread: Sleeping 1s before next retry. 17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001 17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0 17:26:12 MainThread: Sleeping 1s before next retry. 17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001 17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0 17:26:12 MainThread: Sleeping 1s before next retry. 17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001 17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0 17:26:12 MainThread: Sleeping 1s before next retry. 17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001 17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0 17:26:12 MainThread: Sleeping 1s before next retry. 17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001 17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0 17:26:12 MainThread: Sleeping 1s before next retry. 17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001 17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0 17:26:12 MainThread: Sleeping 1s before next retry. 17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001 17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0 17:26:12 MainThread: Sleeping 1s before next retry. 17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001 17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0 17:26:12 MainThread: Sleeping 1s before next retry. 17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001 17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0 17:26:12 MainThread: Sleeping 1s before next retry. 17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001 17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0 17:26:12 MainThread: Sleeping 1s before next retry. 17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001 17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0 17:26:12 MainThread: Sleeping 1s before next retry. 17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001 17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0 17:26:12 MainThread: Sleeping 1s before next retry. 17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001 17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0 17:26:12 MainThread: Sleeping 1s before next retry. 17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001 17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0 17:26:12 MainThread: Sleeping 1s before next retry. 17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001 17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0 17:26:12 MainThread: Sleeping 1s before next retry. 17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001 17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0 17:26:12 MainThread: Sleeping 1s before next retry. 17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001 17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0 17:26:12 MainThread: Sleeping 1s before next retry. 17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001 17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0 17:26:12 MainThread: Sleeping 1s before next retry. 17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001 17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0 17:26:12 MainThread: Sleeping 1s before next retry. 17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001 17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0 17:26:12 MainThread: Sleeping 1s before next retry. 17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001 17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0 17:26:12 MainThread: Sleeping 1s before next retry. 17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001 17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0 17:26:12 MainThread: Sleeping 1s before next retry. 17:26:12 MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-boost-static-burst-slave-06e9.vpc.cloudera.com:25001 17:26:12 MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=1. Current value: 0 17:26:12 MainThread: Sleeping 1s before next retry. 17:26:12 generated xml file: /data/jenkins/workspace/impala-external-gerrit-verify-merge-ASF/repos/Impala/logs/ee_tests/results/TEST-impala-serial.xml 17:26:12 =========================== short test summary info ============================
It doesn't make sense that it's waiting for the number to go to 1. This is probably because it looks at the current state of the Impala daemon to figure out what the "initial" value is, but we don't make any effort to wait for all fragments to finish before starting the test.