[IMPALA-6351] TestValidateMetrics failing for rhel tests

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Not A Bug
• Affects Version/s: Impala 2.12.0
• Component/s: Backend
• Labels: flaky
• Target Version: Impala 2.12.0
• Epic Color: ghx-label-2
• Assignee: Bikramjeet Vig
• Reporter: Vuk Ercegovac
• Created: 22/Dec/17 22:45
• Updated: 08/Mar/18 19:26
• Resolved: 08/Mar/18 19:26

Description

On a recent rhel exhaustive test (kicked off on 12/21):

verifiers/test_verify_metrics.py:34: in test_metrics_are_zero
verifier.verify_metrics_are_zero()
verifiers/metric_verifier.py:47: in verify_metrics_are_zero
self.wait_for_metric(metric, 0, timeout)
verifiers/metric_verifier.py:62: in wait_for_metric
self.impalad_service.wait_for_metric_value(metric_name, expected_value, timeout)
common/impala_service.py:89: in wait_for_metric_value
(metric_name, expected_value, timeout)
E AssertionError: Metric value impala-server.mem-pool.total-bytes did not reach value 0 in 60s

Here is the metric that was obtained:

...
MainThread: Getting metric: impala-server.mem-pool.total-bytes from localhost:25000
MainThread: Waiting for metric value 'impala-server.mem-pool.total-bytes'=0. Current value: 3290112
MainThread: Sleeping 1s before next retry.
MainThread: Getting metric: impala-server.mem-pool.total-bytes from localhost:25000
MainThread: Waiting for metric value 'impala-server.mem-pool.total-bytes'=0. Current value: 3290112
MainThread: Sleeping 1s before next retry.

Similarly, the number of unused buffers did not reach the expected value:
verifiers/test_verify_metrics.py:39: in test_num_unused_buffers
verifier.verify_num_unused_buffers()
verifiers/metric_verifier.py:59: in verify_num_unused_buffers
timeout)
verifiers/metric_verifier.py:62: in wait_for_metric
self.impalad_service.wait_for_metric_value(metric_name, expected_value, timeout)
common/impala_service.py:89: in wait_for_metric_value
(metric_name, expected_value, timeout)
E AssertionError: Metric value impala-server.io-mgr.num-unused-buffers did not reach value 204 in 60s

And the value that was observed:
....
MainThread: Getting metric: impala-server.io-mgr.num-unused-buffers from localhost:25000
MainThread: Waiting for metric value 'impala-server.io-mgr.num-unused-buffers'=204. Current value: 203
MainThread: Sleeping 1s before next retry.
MainThread: Getting metric: impala-server.io-mgr.num-unused-buffers from localhost:25000
MainThread: Waiting for metric value 'impala-server.io-mgr.num-unused-buffers'=204. Current value: 203
MainThread: Sleeping 1s before next retry.
MainThread: Getting metric: impala-server.io-mgr.num-unused-buffers from localhost:25000
MainThread: Waiting for metric value 'impala-server.io-mgr.num-unused-buffers'=204. Current value: 203
MainThread: Sleeping 1s before next retry.

Issue Links

• is duplicated by: IMPALA-6443 Memory leak on exhaustive rhel7 test runs
• is duplicated by: IMPALA-6352 TestTableSample took too long in recent tests
• is related to: IMPALA-6352 TestTableSample took too long in recent tests