IMPALA-2736 there was an increase in peak memory consumption mostly due to the Parquet scanner.
In most cases the Parquet scanner ends up buffering more batches than needed.
In the attached profile the scanner memory increases from 2.17GB to 3.3GB.
The following query options may help to reduce scanner memory consumption:
- Reduce the number of scanner threads (set num_scanner_threads=30)
- Reduce the batch size (set batch_size=512)
Of course, increasing the mem limit may also help.