[IMPALA-3205] Validate and fix spilling performance and memory usage of new buffer pool

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: Impala 2.6.0
• Fix Version/s: Impala 2.10.0
• Component/s: Backend
• Labels: resource-management
• Target Version: Impala 2.10.0
• Assignee: Tim Armstrong
• Reporter: Tim Armstrong
• Created: 18/Mar/16 00:26
• Updated: 18/Aug/17 21:33
• Resolved: 18/Aug/17 21:33

Description

We want to ensure that performance doesn't regress:

• Memory usage for the new buffer pool (with the old small buffers optimisation removed) should be similar or less than the old code for typical workloads. This will require reducing the buffer size - e.g. to 1MB or less.
• Spill-to-disk performance should not regress more than is acceptable. This may be tricky to get right since the I/O size will be reduced from 8MB. We may need to allocate logically sequential blocks in contiguous disk and do some kind of prefetching in the buffer pool to issue multiple writes at a time.