Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
-
ghx-label-7
Description
In some cases pre-aggregations can balloon up and consume lots of memory, forcing the merge aggregation to spill. This is often OK as long as the preaggregation is reducing the input sufficiently, since it reduces the amount of data shuffled over the network.
However in some cases it's preferable to be more conservative with memory and just cap the size of the preaggregation to prevent it ballooning too much.
It would be useful to add a query option to directly limit the memory consumption of the preaggregations.
Attachments
Issue Links
- is related to
-
IMPALA-9591 Automatically cap pre-aggregation memory consumption to match estimate
- Open
-
IMPALA-9765 Preaggregations should stop expanding if memory is scarce
- Open