Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-4487

stress test occasionally leaves runners lingering

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: Impala 2.8.0
    • Fix Version/s: None
    • Component/s: Infrastructure
    • Labels:

      Description

      The stress test runner concurrent_select.py sometimes continues to run after all the requested queries have been executed. This will be evident from the console report: the number of Done queries will be the number of queries set by concurrent_select.py --max-queries but concurrent_select.py will just continue to run indefinitely until terminated.

      I've looked at a cluster at this state and can't evidence of any queries in flight or hung. It leads me to believe then that the bug lies in the test infrastructure, not Impala.

      The debug logs show this over and over:

      08:37:53 2845 140608723109632 DEBUG:concurrent_select[398]:Producer is alive: False
      08:37:53 2845 140608723109632 DEBUG:concurrent_select[399]:Consumer is alive: False
      08:37:53 2845 140608723109632 DEBUG:concurrent_select[400]:Queue size: 0
      08:37:53 2845 140608723109632 DEBUG:concurrent_select[401]:Runners: 1
      

      Workaround:
      Send SIGTERM to the hung child process.

        Attachments

        1. stacks.txt
          11 kB
          Michael Brown

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              mikeb Michael Brown
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated: