The small buffer optimization patch, which includes also a fix for
IMPALA-2330, exposed issues with the buffered-tuple-stream, buffered-block-mgr and how PAGG and PHJ handle various cases.
If a stream is using small buffers and those buffers get full, then the next call to Stream::AllocateRow() by PAGG will fail, because we do not auto-switch to IO-buffers. We would deal with this failure with a hammer: by spilling a partition and switching to IO-buffers all streams. Then, in the next partitioning level we would start all the partitions with IO-buffers (no small buffers) and we would bypass this problem.
The less harmful but important result of this issue is excessive/pre-mature spilling. That is, we end up spilling partitions of PAGG just because the small buffers one of the streams was ending up full, and even though there is plenty of memory available. The most harmful result is queries failing with OOM because of the additional memory needed for the unneeded repartitions.