[IMPALA-6241] Timeout in TestAdmissionControllerStress.test_mem_limit under ASAN

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: Impala 2.11.0
• Fix Version/s: Impala 2.11.0
• Labels: broken-build
• Epic Color: ghx-label-3
• Assignee: Tim Armstrong
• Reporter: Tianyi Wang
• Created: 27/Nov/17 18:56
• Updated: 26/Mar/19 18:55
• Resolved: 29/Nov/17 17:31

Description

In an ASAN build, test_mem_limit times out in end_admitted_queries(). It's probably related to IMPALA-6227

    def end_admitted_queries(self, num_queries):
      """
        Requests each admitted query to end its query.
        """
      assert len(self.executing_threads) >= num_queries
      LOG.debug("Requesting {0} clients to end queries".format(num_queries))
    
      # Request admitted clients to end their queries
      current_executing_queries = []
      for i in xrange(num_queries):
        # pop() is thread-safe, it's OK if another thread is appending concurrently.
        thread = self.executing_threads.pop(0)
        LOG.debug("Cancelling query %s", thread.query_num)
        assert thread.query_state == 'ADMITTED'
        current_executing_queries.append(thread)
        thread.query_state = 'REQUEST_QUERY_END'
    
      # Wait for the queries to end
      start_time = time()
      while True:
        all_done = True
        for thread in self.all_threads:
          if thread.query_state == 'REQUEST_QUERY_END':
            all_done = False
        if all_done:
          break
>       assert (time() - start_time < STRESS_TIMEOUT),\
          "Timed out waiting %s seconds for query end" % (STRESS_TIMEOUT,)
E       AssertionError: Timed out waiting 60 seconds for query end
E       assert (1511608615.681211 - 1511608555.6205289) < 60
E        +  where 1511608615.681211 = time()
custom_cluster/test_admission_controller.py:623: AssertionError

Issue Links

• is broken by: IMPALA-1575 Cancelled queries do not yield resources until close
• is related to: IMPALA-8295 test_mem_limit in test_admission_controller timed out waiting for query to end
• relates to: IMPALA-6227 TestAdmissionControllerStress can be flaky