Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-2990

Coordinator should timeout and cancel queries with unresponsive / stuck executors

    XMLWordPrintableJSON

Details

    Description

      The coordinator currently waits indefinitely if it does not hear back from a backend. This could cause a query to hang indefinitely in case of a network error, etc.

      We should add logic for determining when a backend is unresponsive and kill the query. The logic should mostly revolve around Coordinator::Wait() and Coordinator::UpdateFragmentExecStatus() based on whether it receives periodic updates from a backed (via FragmentExecState::ReportStatusCb()).

      Attachments

        Issue Links

          blocks

          Sub-task - The sub-task of the issue IMPALA-6787 On large secure clusters the connection setup thread becomes bottleneck at warmup and cause occasional timeout failures

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. IMPALA-6338 Tests fail due to runtime profile for query with limit missing pieces

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          incorporates

          Sub-task - The sub-task of the issue IMPALA-4555 Don't cancel query for failed ReportExecStatus (done=false) RPC

          • Major - Major loss of function.
          • Resolved
          is depended upon by

          Sub-task - The sub-task of the issue IMPALA-5119 Don't make RPCs from Coordinator::UpdateBackendExecStatus()

          • Major - Major loss of function.
          • Open
          is duplicated by

          Improvement - An improvement or enhancement to an existing feature or task. IMPALA-414 Impala server cannot detect crash-restart failures reliably

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          is related to

          Sub-task - The sub-task of the issue IMPALA-4063 Make fragment instance reports per-query (or per-host) instead of per-fragment instance.

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. IMPALA-6596 Query failed with OOM on coordinator while remote fragments on other nodes continue to run

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. IMPALA-9919 Bad Impala Performance after a period of time

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. IMPALA-2567 KRPC milestone 1

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. IMPALA-5576 Wrong Cancel() in QueryState::ReportExecStatusAux() can lead to coordinator hang

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. IMPALA-5746 Remote fragments continue to hold onto memory after stopping the coordinator daemon

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. IMPALA-8327 TestRPCTimeout::test_reportexecstatus_retry() times out on exhaustive build

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. IMPALA-3160 Queries may not get cancelled if cancellation pool hits MAX_CANCELLATION_QUEUE_SIZE

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. IMPALA-539 Impala should gather final runtime profile from fragments for aborted/cancelled query

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          Parent Feature

          Sub-task - The sub-task of the issue IMPALA-3380 Add TCP timeouts to all RPCs that don't block

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          relates to

          Sub-task - The sub-task of the issue IMPALA-6984 Coordinator should cancel backends when returning EOS

          • Major - Major loss of function.
          • Reopened
          requires

          Sub-task - The sub-task of the issue IMPALA-7163 Implement a state machine for the QueryState class

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              twmarshall Thomas Tauber-Marshall
              sailesh Sailesh Mukil
              Votes:
              2 Vote for this issue
              Watchers:
              20 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: