[IMPALA-4487] stress test occasionally leaves runners lingering

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Cannot Reproduce
• Affects Version/s: Impala 2.8.0
• Component/s: Infrastructure
• Labels: stress
• Target Version: Product Backlog
• Assignee: Unassigned
• Reporter: Michael Brown
• Created: 15/Nov/16 17:11
• Updated: 19/Jun/20 20:22
• Resolved: 19/Jun/20 20:22

Description

The stress test runner concurrent_select.py sometimes continues to run after all the requested queries have been executed. This will be evident from the console report: the number of Done queries will be the number of queries set by concurrent_select.py --max-queries but concurrent_select.py will just continue to run indefinitely until terminated.

I've looked at a cluster at this state and can't evidence of any queries in flight or hung. It leads me to believe then that the bug lies in the test infrastructure, not Impala.

The debug logs show this over and over:

08:37:53 2845 140608723109632 DEBUG:concurrent_select[398]:Producer is alive: False
08:37:53 2845 140608723109632 DEBUG:concurrent_select[399]:Consumer is alive: False
08:37:53 2845 140608723109632 DEBUG:concurrent_select[400]:Queue size: 0
08:37:53 2845 140608723109632 DEBUG:concurrent_select[401]:Runners: 1

Workaround:
Send SIGTERM to the hung child process.