[IMPALA-9124] Transparently retry queries that fail due to cluster membership changes

• Type: New Feature
• Status: In Progress
• Priority: Critical
• Resolution: Unresolved
• Component/s: Backend, Clients
• Epic Color: ghx-label-5
• Assignee: Sahil Takiar
• Reporter: Sahil Takiar
• Created: 05/Nov/19 17:09
• Updated: 02/May/23 16:13

Description

Currently, if the Impala Coordinator or any Executors run into errors during query execution, Impala will fail the entire query. It would improve user experience to transparently retry the query for some transient, recoverable errors.

This JIRA focuses on retrying queries that would otherwise fail due to cluster membership changes. Specifically, node failures that cause changes in the cluster membership (currently the Coordinator cancels all queries running on a node if it detects that the node is no longer part of the cluster) and node blacklisting (the Coordinator blacklists a node because it detects a problem with that node - can’t execute RPCs against the node). It is not focused on retrying general errors (e.g. any frontend errors, MemLimitExceeded exceptions, etc.).

Issue Links

• is related to: IMPALA-6984 Coordinator should cancel backends when returning EOS
• is related to: IMPALA-8339 Coordinator should be more resilient to fragment instances startup failure
• is related to: IMPALA-9834 test_query_retries.TestQueryRetries is flaky on erasure coding configurations
• is related to: IMPALA-9113 Queries can hang if an impalad is killed after a query has FINISHED
• is related to: IMPALA-10585 retry_failed_queries=true should not apply to DMLs
• is related to: IMPALA-6194 Ensure all fragment instances notice cancellation
• is related to: IMPALA-8138 Re-introduce rpc debugging options
• is related to: IMPALA-2638 Retry queries that fail during scheduling
• relates to: IMPALA-9299 Node Blacklisting: Coordinators should blacklist unhealthy nodes
• requires: IMPALA-6894 Use an internal representation of query states in ClientRequestState

Sub-Tasks

• Classify certain errors as retryable: Resolved
• Add support for single query retries on cluster membership changes: Resolved
• Avoid copying TExecRequest when retrying queries: Resolved
• Client logs should indicate if a query has been retried: Resolved
• Query progress bar freezes when a query is retried: Resolved
• ACID-query retry integration: Resolved
• Link failed and retried runtime profiles: Resolved
• Retried queries that blacklist nodes should ensure they don't run on the blacklisted node: Resolved
• Retryable queries should spool all results before returning any to the client: Resolved
• TSAN data race in QueryDriver::CreateRetriedClientRequestState: Resolved
• TSAN lock-order-inversion warning in QueryDriver::RetryQueryFromThread: Resolved
• Hit DCHECK when retrying a query in FINISHED state: Resolved
• summary and profile command in impala-shell should show both original and retried info: Resolved
• Fix error reporting when AuxErrorInfoPB is present without an error: Open
• Test coverage for query retries when there is a network partition: Open
• Retried runtime profile should include some information about previous query attempts: Open
• Add impalad level metrics for query retries: Open
• Queries should only be retried if all fragments fail with retryable errors: Open
• Re-factor ImpalaServer, ClientRequestState, Coordinator protocol: Open
• Test that queries are not retried if they cause an impalad to crash: Open
• Web UI improvements for retried queries: Open
• Add support for multi query retries on cluster membership changes: Open
• Profile log does not include profiles of failed queries: Open
• Impala Doc: Add docs for transparent query retries: Open
• Consider using num_rows_fetched instead of fetched_rows in checking whether client has fetched any results in TryQueryRetry: Open