Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-5488

TestValidateMetrics fails sometimes when running tests on Isilon

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: Impala 2.10.0
    • Fix Version/s: Impala 2.10.0
    • Component/s: Backend
    • Labels:

      Description

      This happened on commit e0f69ca1da448246a6260099c64f4cc5862188a2. I'm unsure when it started failing, because it was masked by IMPALA-5423. The last green Isilon build was 014c5603f867907963f3821948f90d526e9a4789, which is before the file handle cache change was merged.

      23:32:26 ============ 1367 passed, 82 skipped, 28 xfailed in 3965.31 seconds ============
      23:32:26 ============================= test session starts ==============================
      23:32:26 platform linux2 -- Python 2.6.6, pytest-2.9.2, py-1.4.32, pluggy-0.3.1 -- /data/jenkins/workspace/impala-umbrella-build-and-test-isilon/repos/Impala/bin/../infra/python/env/bin/python
      23:32:26 cachedir: .cache
      23:32:26 rootdir: /data/jenkins/workspace/impala-umbrella-build-and-test-isilon/repos/Impala/tests, inifile: pytest.ini
      23:32:26 plugins: random-0.2, xdist-1.15.0
      23:32:26 collecting ... collected 2 items
      23:32:26 
      23:32:26 verifiers/test_verify_metrics.py::TestValidateMetrics::test_metrics_are_zero FAILED
      23:32:26 verifiers/test_verify_metrics.py::TestValidateMetrics::test_num_unused_buffers PASSED
      23:32:26 
      23:32:26  generated xml file: /data/jenkins/workspace/impala-umbrella-build-and-test-isilon/repos/Impala/logs/ee_tests/results/TEST-impala-verify-metrics.xml 
      23:32:26 =========================== short test summary info ============================
      23:32:26 FAIL verifiers/test_verify_metrics.py::TestValidateMetrics::()::test_metrics_are_zero
      23:32:26 =================================== FAILURES ===================================
      23:32:26 __________________ TestValidateMetrics.test_metrics_are_zero ___________________
      23:32:26 verifiers/test_verify_metrics.py:34: in test_metrics_are_zero
      23:32:26     verifier.verify_metrics_are_zero()
      23:32:26 verifiers/metric_verifier.py:44: in verify_metrics_are_zero
      23:32:26     self.wait_for_metric(metric, 0, timeout)
      23:32:26 verifiers/metric_verifier.py:59: in wait_for_metric
      23:32:26     self.impalad_service.wait_for_metric_value(metric_name, expected_value, timeout)
      23:32:26 common/impala_service.py:89: in wait_for_metric_value
      23:32:26     (metric_name, expected_value, timeout)
      23:32:26 E   AssertionError: Metric value impala-server.io-mgr.num-open-files did not reach value 0 in 60s
      23:32:26 ----------------------------- Captured stderr call -----------------------------
      23:32:26 MainThread: Getting metric: impala-server.hash-table.total-bytes from localhost:25000
      23:32:26 MainThread: Metric 'impala-server.hash-table.total-bytes' has reach desired value: 0
      23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000
      23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2
      23:32:26 MainThread: Sleeping 1s before next retry.
      23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000
      23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2
      23:32:26 MainThread: Sleeping 1s before next retry.
      23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000
      23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2
      23:32:26 MainThread: Sleeping 1s before next retry.
      23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000
      23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2
      23:32:26 MainThread: Sleeping 1s before next retry.
      23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000
      23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2
      23:32:26 MainThread: Sleeping 1s before next retry.
      23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000
      23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2
      23:32:26 MainThread: Sleeping 1s before next retry.
      23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000
      23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2
      23:32:26 MainThread: Sleeping 1s before next retry.
      23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000
      23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2
      23:32:26 MainThread: Sleeping 1s before next retry.
      23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000
      23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2
      23:32:26 MainThread: Sleeping 1s before next retry.
      23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000
      23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2
      23:32:26 MainThread: Sleeping 1s before next retry.
      23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000
      23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2
      23:32:26 MainThread: Sleeping 1s before next retry.
      23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000
      23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2
      23:32:26 MainThread: Sleeping 1s before next retry.
      23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000
      23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2
      23:32:26 MainThread: Sleeping 1s before next retry.
      23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000
      23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2
      23:32:26 MainThread: Sleeping 1s before next retry.
      23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000
      23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2
      23:32:26 MainThread: Sleeping 1s before next retry.
      23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000
      23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2
      23:32:26 MainThread: Sleeping 1s before next retry.
      23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000
      23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2
      23:32:26 MainThread: Sleeping 1s before next retry.
      23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000
      23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2
      23:32:26 MainThread: Sleeping 1s before next retry.
      23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000
      23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2
      23:32:26 MainThread: Sleeping 1s before next retry.
      23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000
      23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2
      23:32:26 MainThread: Sleeping 1s before next retry.
      23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000
      23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2
      23:32:26 MainThread: Sleeping 1s before next retry.
      23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000
      23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2
      23:32:26 MainThread: Sleeping 1s before next retry.
      23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000
      23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2
      23:32:26 MainThread: Sleeping 1s before next retry.
      23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000
      23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2
      23:32:26 MainThread: Sleeping 1s before next retry.
      23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000
      23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2
      23:32:26 MainThread: Sleeping 1s before next retry.
      23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000
      23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2
      23:32:26 MainThread: Sleeping 1s before next retry.
      23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000
      23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2
      23:32:26 MainThread: Sleeping 1s before next retry.
      23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000
      23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2
      23:32:26 MainThread: Sleeping 1s before next retry.
      23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000
      23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2
      23:32:26 MainThread: Sleeping 1s before next retry.
      23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000
      23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2
      23:32:26 MainThread: Sleeping 1s before next retry.
      23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000
      23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2
      23:32:26 MainThread: Sleeping 1s before next retry.
      23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000
      23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2
      23:32:26 MainThread: Sleeping 1s before next retry.
      23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000
      23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2
      23:32:26 MainThread: Sleeping 1s before next retry.
      23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000
      23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2
      23:32:26 MainThread: Sleeping 1s before next retry.
      23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000
      23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2
      23:32:26 MainThread: Sleeping 1s before next retry.
      23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000
      23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2
      23:32:26 MainThread: Sleeping 1s before next retry.
      23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000
      23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2
      23:32:26 MainThread: Sleeping 1s before next retry.
      23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000
      23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2
      23:32:26 MainThread: Sleeping 1s before next retry.
      23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000
      23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2
      23:32:26 MainThread: Sleeping 1s before next retry.
      23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000
      23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2
      23:32:26 MainThread: Sleeping 1s before next retry.
      23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000
      23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2
      23:32:26 MainThread: Sleeping 1s before next retry.
      23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000
      23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2
      23:32:26 MainThread: Sleeping 1s before next retry.
      23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000
      23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2
      23:32:26 MainThread: Sleeping 1s before next retry.
      23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000
      23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2
      23:32:26 MainThread: Sleeping 1s before next retry.
      23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000
      23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2
      23:32:26 MainThread: Sleeping 1s before next retry.
      23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000
      23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2
      23:32:26 MainThread: Sleeping 1s before next retry.
      23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000
      23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2
      23:32:26 MainThread: Sleeping 1s before next retry.
      23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000
      23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2
      23:32:26 MainThread: Sleeping 1s before next retry.
      23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000
      23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2
      23:32:26 MainThread: Sleeping 1s before next retry.
      23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000
      23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2
      23:32:26 MainThread: Sleeping 1s before next retry.
      23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000
      23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2
      23:32:26 MainThread: Sleeping 1s before next retry.
      23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000
      23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2
      23:32:26 MainThread: Sleeping 1s before next retry.
      23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000
      23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2
      23:32:26 MainThread: Sleeping 1s before next retry.
      23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000
      23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2
      23:32:26 MainThread: Sleeping 1s before next retry.
      23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000
      23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2
      23:32:26 MainThread: Sleeping 1s before next retry.
      23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000
      23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2
      23:32:26 MainThread: Sleeping 1s before next retry.
      23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000
      23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2
      23:32:26 MainThread: Sleeping 1s before next retry.
      23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000
      23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2
      23:32:26 MainThread: Sleeping 1s before next retry.
      23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000
      23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2
      23:32:26 MainThread: Sleeping 1s before next retry.
      23:32:26 MainThread: Getting metric: impala-server.io-mgr.num-open-files from localhost:25000
      23:32:26 MainThread: Waiting for metric value 'impala-server.io-mgr.num-open-files'=0. Current value: 2
      23:32:26 MainThread: Sleeping 1s before next retry.
      

        Attachments

          Activity

            People

            • Assignee:
              joemcdonnell Joe McDonnell
              Reporter:
              tarmstrong Tim Armstrong
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: