[IMPALA-6956] check_num_executing fails test_query_expiration

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: Impala 3.1.0
• Fix Version/s: Impala 2.13.0, Impala 3.1.0
• Component/s: Backend
• Labels: broken-build flaky
• Epic Color: ghx-label-5
• Assignee: Vuk Ercegovac
• Reporter: Vuk Ercegovac
• Created: 01/May/18 01:06
• Updated: 07/Jun/18 00:24
• Resolved: 06/Jun/18 15:52

Description

This happened recently on an s3 build. The check that asserts the number of currently running queries comes back a few queries short.

custom_cluster/test_query_expiration.py:71: in test_query_expiration self._check_num_executing(impalad, len(handles)) custom_cluster/test_query_expiration.py:38: in _check_num_executing assert actual == expected 
E assert 2 == 4

The last query is executed with: 

I0426 01:57:29.507251 12229 query-state.cc:395] Executing instance. instance_id=be4bd338f1d8a8ef:651e7a3c00000000 fragment_idx=0 per_fragment_instance_idx=0 coord_state_idx=0 #in-flight=4

So there are 4 in flight at that time. However, prior to this, I see that some are in the process of being cancelled:

I0426 01:57:29.022300 11843 impala-server.cc:1908] Expiring query 524c239e0d9f6356:b70c252400000000 due to execution time limit of 1s.