[IMPALA-7736] stress: AssertionError: The number of executing queries is negative

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Won't Fix
• Affects Version/s: Impala 3.1.0
• Component/s: Infrastructure
• Labels: stress
• Target Version: Impala 3.1.0
• Epic Color: ghx-label-2
• Assignee: Michael Brown
• Reporter: Michael Brown
• Created: 22/Oct/18 18:09
• Updated: 30/Nov/18 16:53
• Resolved: 30/Nov/18 16:53

Description

I'd heard about this anecdotally but haven't seen it until now:

Done | Active | Executing | Mem Lmt Ex | AC Reject | AC Timeout | Time Out | Cancel | Err | Incorrect | Next Qry Mem Lmt | Tot Qry Mem Lmt | Tracked Mem | RSS Mem
 263 |      6 |         3 |         20 |         0 |          0 |        0 |     17 |   0 |         0 |             9818 |           26802 |       15656 |   19329
 263 |      6 |         3 |         20 |         0 |          0 |        0 |     17 |   0 |         0 |             9818 |           26802 |       15534 |   19341
 263 |      6 |         3 |         20 |         0 |          0 |        0 |     17 |   0 |         0 |             9818 |           26802 |       15462 |   19190
 265 |      4 |         1 |         20 |         0 |          0 |        0 |     17 |   0 |         0 |             9818 |           22938 |       15482 |   19565
Traceback (most recent call last):
  File "tests/stress/concurrent_select.py", line 2320, in <module>
    main()
  File "tests/stress/concurrent_select.py", line 2316, in main
    select_probability=args.select_probability)
  File "tests/stress/concurrent_select.py", line 529, in run_queries
    self._wait_for_test_to_finish(impala, should_print_status)
  File "tests/stress/concurrent_select.py", line 943, in _wait_for_test_to_finish
    self._print_status(print_header=(lines_printed == 0))
  File "tests/stress/concurrent_select.py", line 837, in _print_status
    self._num_queries_executing,
  File "tests/stress/concurrent_select.py", line 690, in _num_queries_executing
    assert num_executing >= 0, "The number of executing queries is negative"
AssertionError: The number of executing queries is negative

I've run many thousand-query stress test runs against 3.1 without hitting this. Where this finally happened was against a 2.12-based build, if that matters.