[IMPALA-9530] Allow limiting memory consumed by preaggregation

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Impala 4.0.0
• Component/s: Backend
• Target Version: Impala 4.0.0
• Epic Color: ghx-label-7
• Assignee: Tim Armstrong
• Reporter: Tim Armstrong
• Created: 17/Mar/20 15:42
• Updated: 20/May/20 16:46
• Resolved: 20/Mar/20 17:37

Description

In some cases pre-aggregations can balloon up and consume lots of memory, forcing the merge aggregation to spill. This is often OK as long as the preaggregation is reducing the input sufficiently, since it reduces the amount of data shuffled over the network.

However in some cases it's preferable to be more conservative with memory and just cap the size of the preaggregation to prevent it ballooning too much.

It would be useful to add a query option to directly limit the memory consumption of the preaggregations.

Issue Links

• is related to: IMPALA-9591 Automatically cap pre-aggregation memory consumption to match estimate
• is related to: IMPALA-9765 Preaggregations should stop expanding if memory is scarce