[IMPALA-1575] Cancelled queries do not yield resources until close

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 2.1, Impala 2.3.0
• Fix Version/s: Impala 2.11.0
• Component/s: Backend
• Labels: query-lifecycle resource-management
• Target Version: Impala 2.11.0
• Assignee: Tim Armstrong
• Reporter: Henry Robinson
• Created: 04/Dec/14 22:27
• Updated: 20/Feb/18 00:35
• Resolved: 20/Nov/17 17:53

Description

A cancelled query (for example due to a timeout) or a query that has reached eos (but not explicitly closed) holds (1) resources on the coordinator fragment, (2) all resources accounted by the admission controller, (3) llama reservations. (However, Llama has been unsupported for CDH 5.5 and beyond, so (3) will no longer apply.) All of these are not released until the query is closed, which may not happen promptly for some clients.

This frequently occurs with Hue. Hue (and some other clients that behave similarly) will not close a query until explicitly closed (in the Hue case this is via a javascript callback sent by the browser when closing the Hue tab). If the query is left unattended (or the Hue tab is on a laptop that is closed, or the browser crashes), the close call is never sent, and while the query will "time out", the cancellation doesn't properly clean up resources.

One way to mitigate this issue in this case is by using the --idle_session_timeout impalad argument to fully close a session and all associated queries after some amount of time (but this is not a workaround that works in all cases).

Issue Links

• breaks: IMPALA-6171 Failure in test_admission_controller "assert metric_deltas['timed-out'] == 0"
• breaks: IMPALA-6241 Timeout in TestAdmissionControllerStress.test_mem_limit under ASAN
• breaks: IMPALA-6227 TestAdmissionControllerStress can be flaky
• is related to: IMPALA-6482 Add query option for query time limit
• is related to: IMPALA-6153 Prevent Coordinator::UpdateFilter() running after query exec resources are released
• relates to: IMPALA-6153 Prevent Coordinator::UpdateFilter() running after query exec resources are released
• relates to: IMPALA-5712 Provide option to cancel queries when in CREATED state