Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-6171

Failure in test_admission_controller "assert metric_deltas['timed-out'] == 0"

    Details

    • Epic Color:
      ghx-label-1

      Description

      Seen in two exhaustive release builds, both at git hash 40ec6d0080638efaf3260672ab54ea4674896c5e:

      05:19:10  TestAdmissionControllerStress.test_admission_controller_with_flags[num_queries: 50 | submission_delay_ms: 50 | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: text/none | round_robin_submission: True] 
      05:19:10 
      05:19:10 self = <test_admission_controller.TestAdmissionControllerStress object at 0x3ed4210>
      05:19:10 vector = <tests.common.test_vector.ImpalaTestVector object at 0x58a3c10>
      05:19:10 
      05:19:10     @pytest.mark.execute_serially
      05:19:10     @CustomClusterTestSuite.with_args(
      05:19:10         impalad_args=impalad_admission_ctrl_flags(max_requests=MAX_NUM_CONCURRENT_QUERIES,
      05:19:10           max_queued=MAX_NUM_QUEUED_QUERIES, pool_max_mem=-1),
      05:19:10         statestored_args=_STATESTORED_ARGS)
      05:19:10     def test_admission_controller_with_flags(self, vector):
      05:19:10       self.pool_name = 'default-pool'
      05:19:10       # The pool has no mem resources set, so submitting queries with huge mem_limits
      05:19:10       # should be fine. This exercises the code that does the per-pool memory
      05:19:10       # accounting (see MemTracker::GetPoolMemReserved()) without actually being throttled.
      05:19:10       self.run_admission_test(vector, {'request_pool': self.pool_name,
      05:19:10 >       'mem_limit': sys.maxint})
      05:19:10 
      05:19:10 custom_cluster/test_admission_controller.py:831: 
      05:19:10 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
      05:19:10 
      05:19:10 self = <test_admission_controller.TestAdmissionControllerStress object at 0x3ed4210>
      05:19:10 vector = <tests.common.test_vector.ImpalaTestVector object at 0x58a3c10>
      05:19:10 additional_query_options = {'mem_limit': 9223372036854775807, 'request_pool': 'default-pool'}
      05:19:10 
      05:19:10     def run_admission_test(self, vector, additional_query_options):
      05:19:10       LOG.debug("Starting test case with parameters: %s", vector)
      05:19:10       self.impalads = self.cluster.impalads
      05:19:10       round_robin_submission = vector.get_value('round_robin_submission')
      05:19:10       submission_delay_ms = vector.get_value('submission_delay_ms')
      05:19:10       if not round_robin_submission:
      05:19:10         self.impalads = [self.impalads[0]]
      .......
      final_metrics = self.get_admission_metrics();
      04:40:28       log_metrics("Final metrics: ", final_metrics, logging.INFO);
      04:40:28       metric_deltas = compute_metric_deltas(final_metrics, initial_metrics,
      04:40:28           final_metrics.keys())
      04:40:28 >     assert metric_deltas['timed-out'] == 0
      04:40:28 E     assert 3 == 0
      04:40:28 
      04:40:28 custom_cluster/test_admission_controller.py:792: AssertionError
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                tarmstrong Tim Armstrong
                Reporter:
                twmarshall Thomas Tauber-Marshall
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: