Details
-
Sub-task
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
Impala 2.10.0
-
ghx-label-8
Description
The new buffer pool code releases memory from aggregations at a later point than the old code: the old code will release the memory partition-by-partition as the output rows are emitted.
This is particularly useful with pre/merge aggregations since the preaggregation will be shrinking at the same time as the merge is growing.
With the new code, if all remaining partitions are in memory we can safely release reservation before we move to the next partition. E.g. in NextPartition() we can have something like:
if (spilled_partitions_.empty()) ReleaseUnusedReservation();
where ReleaseUnusedReservation() will release any unused reservation above the minimum back to the query reservation.
A similar problem exists for sorts.