Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-2990

Coordinator should timeout and cancel queries with unresponsive / stuck executors

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

      The coordinator currently waits indefinitely if it does not hear back from a backend. This could cause a query to hang indefinitely in case of a network error, etc.

      We should add logic for determining when a backend is unresponsive and kill the query. The logic should mostly revolve around Coordinator::Wait() and Coordinator::UpdateFragmentExecStatus() based on whether it receives periodic updates from a backed (via FragmentExecState::ReportStatusCb()).

      Attachments

        Issue Links

        blocks

        Sub-task - The sub-task of the issue IMPALA-6787 On large secure clusters the connection setup thread becomes bottleneck at warmup and cause occasional timeout failures

        • Major - Major loss of function.
        • Resolved

        Bug - A problem which impairs or prevents the functions of the product. IMPALA-6338 Tests fail due to runtime profile for query with limit missing pieces

        • Critical - Crashes, loss of data, severe memory leak.
        • Resolved
        incorporates

        Sub-task - The sub-task of the issue IMPALA-4555 Don't cancel query for failed ReportExecStatus (done=false) RPC

        • Major - Major loss of function.
        • Resolved
        is depended upon by

        Sub-task - The sub-task of the issue IMPALA-5119 Don't make RPCs from Coordinator::UpdateBackendExecStatus()

        • Major - Major loss of function.
        • Open
        is duplicated by

        Improvement - An improvement or enhancement to an existing feature or task. IMPALA-414 Impala server cannot detect crash-restart failures reliably

        • Minor - Minor loss of function, or other problem where easy workaround is present.
        • Resolved
        is related to

        Sub-task - The sub-task of the issue IMPALA-4063 Make fragment instance reports per-query (or per-host) instead of per-fragment instance.

        • Major - Major loss of function.
        • Resolved

        Bug - A problem which impairs or prevents the functions of the product. IMPALA-6596 Query failed with OOM on coordinator while remote fragments on other nodes continue to run

        • Major - Major loss of function.
        • Open

        Bug - A problem which impairs or prevents the functions of the product. IMPALA-9919 Bad Impala Performance after a period of time

        • Major - Major loss of function.
        • Open

        Bug - A problem which impairs or prevents the functions of the product. IMPALA-2567 KRPC milestone 1

        • Critical - Crashes, loss of data, severe memory leak.
        • Resolved

        Bug - A problem which impairs or prevents the functions of the product. IMPALA-5576 Wrong Cancel() in QueryState::ReportExecStatusAux() can lead to coordinator hang

        • Critical - Crashes, loss of data, severe memory leak.
        • Resolved

        Bug - A problem which impairs or prevents the functions of the product. IMPALA-5746 Remote fragments continue to hold onto memory after stopping the coordinator daemon

        • Critical - Crashes, loss of data, severe memory leak.
        • Resolved

        Bug - A problem which impairs or prevents the functions of the product. IMPALA-8327 TestRPCTimeout::test_reportexecstatus_retry() times out on exhaustive build

        • Critical - Crashes, loss of data, severe memory leak.
        • Resolved

        Bug - A problem which impairs or prevents the functions of the product. IMPALA-3160 Queries may not get cancelled if cancellation pool hits MAX_CANCELLATION_QUEUE_SIZE

        • Minor - Minor loss of function, or other problem where easy workaround is present.
        • Resolved

        Improvement - An improvement or enhancement to an existing feature or task. IMPALA-539 Impala should gather final runtime profile from fragments for aborted/cancelled query

        • Minor - Minor loss of function, or other problem where easy workaround is present.
        • Resolved
        Parent Feature

        Sub-task - The sub-task of the issue IMPALA-3380 Add TCP timeouts to all RPCs that don't block

        • Minor - Minor loss of function, or other problem where easy workaround is present.
        • Resolved
        relates to

        Sub-task - The sub-task of the issue IMPALA-6984 Coordinator should cancel backends when returning EOS

        • Major - Major loss of function.
        • Reopened
        requires

        Sub-task - The sub-task of the issue IMPALA-7163 Implement a state machine for the QueryState class

        • Major - Major loss of function.
        • Resolved

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            twmarshall Thomas Tauber-Marshall
            sailesh Sailesh Mukil
            Votes:
            2 Vote for this issue
            Watchers:
            20 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment