[IMPALA-5677] Consider ways to reduce the accumulation of clean pages when executing large spilling queries

• Type: Sub-task
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 2.10.0
• Fix Version/s: Impala 2.10.0
• Component/s: Backend
• Labels: resource-management
• Target Version: Impala 2.10.0
• Epic Color: ghx-label-6
• Assignee: Tim Armstrong
• Reporter: Tim Armstrong
• Created: 18/Jul/17 15:40
• Updated: 18/Aug/17 05:54
• Resolved: 18/Aug/17 05:54

Description

By design, the new buffer pool aggressively uses available memory to hold onto spilled data in memory.

On some stress tests runs (and in other testing) I've seen the clean pages memory blow up and cause Impala to use more memory than one might expect based on the query memory limits (but not more than the process limit, so we're not going outside of the configured resource constraints).

I think this may hurt performance more than it helps in many cases because it will squeeze out the OS buffer cache.