[IMPALA-6460] More flexible memory-based admission control policies

• Type: Epic
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 2.11.0
• Fix Version/s: Impala 4.0.0
• Component/s: Distributed Exec
• Labels: admission-control
• Epic Name: Memory-based admission control policies
• Target Version: Product Backlog
• Epic Color: ghx-label-8
• Assignee: Unassigned
• Reporter: Tim Armstrong
• Created: 31/Jan/18 00:13
• Updated: 21/Dec/20 19:24
• Resolved: 21/Dec/20 19:24

Description

Currently there are only two ways to decided how much memory to reserve for each query in memory-based admission control:

• Using the memory estimates, which often makes bad decisions (e.g. huge overestimates) and doesn't have any enforcement on the backend
• Using a static pool or user-set mem_limit, which is very difficult to set to a reasonable value for all queries.

The memory reservation work will allow us to come up with more powerful and flexible polices

Issue Links

• is blocked by: IMPALA-4835 HDFS scans should operate with a constrained number of I/O buffers
• is depended upon by: IMPALA-2945 Pre-aggregation cardinality estimates do not take into account data distribution
• is related to: IMPALA-6856 High memory estimates in Impala 2.11
• is related to: IMPALA-6847 Consider adding workaround for high memory estimates in admission control
• relates to: IMPALA-4834 All operators should operate within a memory constraint