[IMPALA-7349] Automatically choose mem_limit based on estimate, clamped to range

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Impala 3.1.0
• Component/s: Backend
• Labels: admission-control resource-management
• Epic Link: Memory-based admission control policies
• Target Version: Impala 3.1.0
• Epic Color: ghx-label-5
• Assignee: Bikramjeet Vig
• Reporter: Tim Armstrong
• Created: 25/Jul/18 22:37
• Updated: 29/Jul/19 10:47
• Resolved: 08/Oct/18 20:28

Description

We should add admission control support for intelligently choosing how much memory to give a query, based on the memory estimate. Currently we require that you set a single mem_limit per pool. Instead it would be better if we allowed configuring min/max guardrails within which admission control chose an amount of memory that will allow good performance.

Initially, I think mem_limit will be the same for all backends. Eventually it could make sense to have different limits per backend depending on the fragments.

Issue Links

• duplicates: IMPALA-8679 Make the query options set in the dynamic resource pool/admission control un-overridable in the user session
• is depended upon by: IMPALA-7389 Admission control should set aside less memory on dedicated coordinator if coordinator fragment is lightweight
• is related to: IMPALA-8418 Restrict user from setting resource limits to high values at session level

Sub-Tasks

• Impala Doc: Doc admission control support for automatically setting per host memory limit for a query: Closed