[IMPALA-7446] Queries can spill earlier than necessary because of accumulation of free buffers and clean pages

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 2.10.0, Impala 2.11.0, Impala 3.0, Impala 2.12.0
• Fix Version/s: Impala 3.2.0
• Component/s: Backend
• Labels: resource-management
• Target Version: Impala 3.2.0
• Epic Color: ghx-label-6
• Assignee: Tim Armstrong
• Reporter: Tim Armstrong
• Created: 14/Aug/18 23:26
• Updated: 09/Jan/19 07:00
• Resolved: 09/Jan/19 06:59

Description

See IMPALA-7442 for an example where the query started to spill even when memory could have been made available by freeing buffers or evicting clean pages. Usually this would just result in spilling earlier than necessary, but in the case of IMPALA-7442 it lead to a query failure.

My original intent was that BufferPool::ReleaseMemory() should be called in situations like this, but that was not done.

Issue Links

• is related to: IMPALA-7442 test_semi_joins_exhaustive occasionally fails
• relates to: IMPALA-8020 Process memory consumption does not include unused reservations