[IMPALA-6171] Failure in test_admission_controller "assert metric_deltas['timed-out'] == 0"

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: Impala 2.11.0
• Fix Version/s: Impala 2.11.0
• Component/s: Infrastructure
• Labels: broken-build
• Epic Color: ghx-label-1
• Assignee: Tim Armstrong
• Reporter: Thomas Tauber-Marshall
• Created: 08/Nov/17 18:05
• Updated: 05/Dec/17 17:37
• Resolved: 08/Nov/17 22:38

Description

Seen in two exhaustive release builds, both at git hash 40ec6d0080638efaf3260672ab54ea4674896c5e:

05:19:10  TestAdmissionControllerStress.test_admission_controller_with_flags[num_queries: 50 | submission_delay_ms: 50 | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: text/none | round_robin_submission: True] 
05:19:10 
05:19:10 self = <test_admission_controller.TestAdmissionControllerStress object at 0x3ed4210>
05:19:10 vector = <tests.common.test_vector.ImpalaTestVector object at 0x58a3c10>
05:19:10 
05:19:10     @pytest.mark.execute_serially
05:19:10     @CustomClusterTestSuite.with_args(
05:19:10         impalad_args=impalad_admission_ctrl_flags(max_requests=MAX_NUM_CONCURRENT_QUERIES,
05:19:10           max_queued=MAX_NUM_QUEUED_QUERIES, pool_max_mem=-1),
05:19:10         statestored_args=_STATESTORED_ARGS)
05:19:10     def test_admission_controller_with_flags(self, vector):
05:19:10       self.pool_name = 'default-pool'
05:19:10       # The pool has no mem resources set, so submitting queries with huge mem_limits
05:19:10       # should be fine. This exercises the code that does the per-pool memory
05:19:10       # accounting (see MemTracker::GetPoolMemReserved()) without actually being throttled.
05:19:10       self.run_admission_test(vector, {'request_pool': self.pool_name,
05:19:10 >       'mem_limit': sys.maxint})
05:19:10 
05:19:10 custom_cluster/test_admission_controller.py:831: 
05:19:10 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
05:19:10 
05:19:10 self = <test_admission_controller.TestAdmissionControllerStress object at 0x3ed4210>
05:19:10 vector = <tests.common.test_vector.ImpalaTestVector object at 0x58a3c10>
05:19:10 additional_query_options = {'mem_limit': 9223372036854775807, 'request_pool': 'default-pool'}
05:19:10 
05:19:10     def run_admission_test(self, vector, additional_query_options):
05:19:10       LOG.debug("Starting test case with parameters: %s", vector)
05:19:10       self.impalads = self.cluster.impalads
05:19:10       round_robin_submission = vector.get_value('round_robin_submission')
05:19:10       submission_delay_ms = vector.get_value('submission_delay_ms')
05:19:10       if not round_robin_submission:
05:19:10         self.impalads = [self.impalads[0]]
.......
final_metrics = self.get_admission_metrics();
04:40:28       log_metrics("Final metrics: ", final_metrics, logging.INFO);
04:40:28       metric_deltas = compute_metric_deltas(final_metrics, initial_metrics,
04:40:28           final_metrics.keys())
04:40:28 >     assert metric_deltas['timed-out'] == 0
04:40:28 E     assert 3 == 0
04:40:28 
04:40:28 custom_cluster/test_admission_controller.py:792: AssertionError

Issue Links

• is broken by: IMPALA-1575 Cancelled queries do not yield resources until close