Details
-
Bug
-
Status: Resolved
-
Blocker
-
Resolution: Fixed
-
Impala 2.10.0
-
ghx-label-2
Description
This happened on commit e0f69ca1da448246a6260099c64f4cc5862188a2. I'm unsure when it started failing, because it was masked by IMPALA-5423. The last green Isilon build was 014c5603f867907963f3821948f90d526e9a4789, which is before the file handle cache change was merged.
23:32:26 ============ 1367 passed, 82 skipped, 28 xfailed in 3965.31 seconds ============ 23:32:26 ============================= test session starts ============================== 23:32:26 platform linux2 -- Python 2.6.6, pytest-2.9.2, py-1.4.32, pluggy-0.3.1 -- /data/jenkins/workspace/impala-umbrella-build-and-test-isilon/repos/Impala/bin/../infra/python/env/bin/python 23:32:26 cachedir: .cache 23:32:26 rootdir: /data/jenkins/workspace/impala-umbrella-build-and-test-isilon/repos/Impala/tests, inifile: pytest.ini 23:32:26 plugins: random-0.2, xdist-1.15.0 23:32:26 collecting ... collected 2 items 23:32:26 23:32:26 verifiers/test_verify_metrics.py::TestValidateMetrics::test_metrics_are_zero FAILED 23:32:26 verifiers/test_verify_metrics.py::TestValidateMetrics::test_num_unused_buffers PASSED 23:32:26 23:32:26 generated xml file: /data/jenkins/workspace/impala-umbrella-build-and-test-isilon/repos/Impala/logs/ee_tests/results/TEST-impala-verify-metrics.xml 23:32:26 =========================== short test summary info ============================ 23:32:26 FAIL verifiers/test_verify_metrics.py::TestValidateMetrics::()::test_metrics_are_zero 23:32:26 =================================== FAILURES =================================== 23:32:26 __________________ TestValidateMetrics.test_metrics_are_zero ___________________ 23:32:26 verifiers/test_verify_metrics.py:34: in test_metrics_are_zero 23:32:26 verifier.verify_metrics_are_zero() 23:32:26 verifiers/metric_verifier.py:44: in verify_metrics_are_zero 23:32:26 self.wait_for_metric(metric, 0, timeout) 23:32:26 verifiers/metric_verifier.py:59: in wait_for_metric 23:32:26 self.impalad_service.wait_for_metric_value(metric_name, expected_value, timeout) 23:32:26 common/impala_service.py:89: in wait_for_metric_value 23:32:26 (metric_name, expected_value, timeout) 23:32:26 E AssertionError: Metric value impala-server.io-mgr.num-open-files did not reach value 0 in 60s 23:32:26 ----------------------------- Captured stderr call ----------------------------- 23:32:26 MainThread: Getting metric: impala-server.hash-table.total-bytes from localhost:25000 23:32:26 MainThread: Metric 'impala-server.hash-table.total-bytes' has reach desired value: 0 23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000 23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2 23:32:26 MainThread: Sleeping 1s before next retry. 23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000 23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2 23:32:26 MainThread: Sleeping 1s before next retry. 23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000 23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2 23:32:26 MainThread: Sleeping 1s before next retry. 23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000 23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2 23:32:26 MainThread: Sleeping 1s before next retry. 23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000 23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2 23:32:26 MainThread: Sleeping 1s before next retry. 23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000 23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2 23:32:26 MainThread: Sleeping 1s before next retry. 23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000 23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2 23:32:26 MainThread: Sleeping 1s before next retry. 23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000 23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2 23:32:26 MainThread: Sleeping 1s before next retry. 23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000 23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2 23:32:26 MainThread: Sleeping 1s before next retry. 23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000 23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2 23:32:26 MainThread: Sleeping 1s before next retry. 23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000 23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2 23:32:26 MainThread: Sleeping 1s before next retry. 23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000 23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2 23:32:26 MainThread: Sleeping 1s before next retry. 23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000 23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2 23:32:26 MainThread: Sleeping 1s before next retry. 23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000 23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2 23:32:26 MainThread: Sleeping 1s before next retry. 23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000 23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2 23:32:26 MainThread: Sleeping 1s before next retry. 23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000 23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2 23:32:26 MainThread: Sleeping 1s before next retry. 23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000 23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2 23:32:26 MainThread: Sleeping 1s before next retry. 23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000 23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2 23:32:26 MainThread: Sleeping 1s before next retry. 23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000 23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2 23:32:26 MainThread: Sleeping 1s before next retry. 23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000 23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2 23:32:26 MainThread: Sleeping 1s before next retry. 23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000 23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2 23:32:26 MainThread: Sleeping 1s before next retry. 23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000 23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2 23:32:26 MainThread: Sleeping 1s before next retry. 23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000 23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2 23:32:26 MainThread: Sleeping 1s before next retry. 23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000 23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2 23:32:26 MainThread: Sleeping 1s before next retry. 23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000 23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2 23:32:26 MainThread: Sleeping 1s before next retry. 23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000 23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2 23:32:26 MainThread: Sleeping 1s before next retry. 23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000 23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2 23:32:26 MainThread: Sleeping 1s before next retry. 23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000 23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2 23:32:26 MainThread: Sleeping 1s before next retry. 23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000 23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2 23:32:26 MainThread: Sleeping 1s before next retry. 23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000 23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2 23:32:26 MainThread: Sleeping 1s before next retry. 23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000 23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2 23:32:26 MainThread: Sleeping 1s before next retry. 23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000 23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2 23:32:26 MainThread: Sleeping 1s before next retry. 23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000 23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2 23:32:26 MainThread: Sleeping 1s before next retry. 23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000 23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2 23:32:26 MainThread: Sleeping 1s before next retry. 23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000 23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2 23:32:26 MainThread: Sleeping 1s before next retry. 23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000 23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2 23:32:26 MainThread: Sleeping 1s before next retry. 23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000 23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2 23:32:26 MainThread: Sleeping 1s before next retry. 23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000 23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2 23:32:26 MainThread: Sleeping 1s before next retry. 23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000 23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2 23:32:26 MainThread: Sleeping 1s before next retry. 23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000 23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2 23:32:26 MainThread: Sleeping 1s before next retry. 23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000 23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2 23:32:26 MainThread: Sleeping 1s before next retry. 23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000 23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2 23:32:26 MainThread: Sleeping 1s before next retry. 23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000 23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2 23:32:26 MainThread: Sleeping 1s before next retry. 23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000 23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2 23:32:26 MainThread: Sleeping 1s before next retry. 23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000 23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2 23:32:26 MainThread: Sleeping 1s before next retry. 23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000 23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2 23:32:26 MainThread: Sleeping 1s before next retry. 23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000 23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2 23:32:26 MainThread: Sleeping 1s before next retry. 23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000 23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2 23:32:26 MainThread: Sleeping 1s before next retry. 23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000 23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2 23:32:26 MainThread: Sleeping 1s before next retry. 23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000 23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2 23:32:26 MainThread: Sleeping 1s before next retry. 23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000 23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2 23:32:26 MainThread: Sleeping 1s before next retry. 23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000 23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2 23:32:26 MainThread: Sleeping 1s before next retry. 23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000 23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2 23:32:26 MainThread: Sleeping 1s before next retry. 23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000 23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2 23:32:26 MainThread: Sleeping 1s before next retry. 23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000 23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2 23:32:26 MainThread: Sleeping 1s before next retry. 23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000 23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2 23:32:26 MainThread: Sleeping 1s before next retry. 23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000 23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2 23:32:26 MainThread: Sleeping 1s before next retry. 23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000 23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2 23:32:26 MainThread: Sleeping 1s before next retry. 23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000 23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2 23:32:26 MainThread: Sleeping 1s before next retry. 23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000 23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2 23:32:26 MainThread: Sleeping 1s before next retry.