[IMPALA-7096] Ensure no memory limit exceeded regressions from IMPALA-4835 because of non-reserved memory

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: Impala 2.13.0, Impala 3.1.0
• Fix Version/s: Impala 3.1.0
• Component/s: Backend
• Labels: resource-management
• Target Version: Impala 3.1.0
• Epic Color: ghx-label-4
• Assignee: Tim Armstrong
• Reporter: Tim Armstrong
• Created: 30/May/18 23:25
• Updated: 10/Oct/23 22:31
• Resolved: 16/Aug/18 21:34

Description

IMPALA-7078 showed some cases where non-buffer memory could accumulate in the row batch queue and cause memory consumption problems.

The decision for whether to spin up a scanner thread in IMPALA-4835 implicitly assumes that buffer memory is the bulk of memory consumed by a scan, but there may be cases where that is not true and the previous heuristic would be more conservative about starting a scanner thread.

We should investigate this further and figure out how to avoid it if there's an issue.

Issue Links

• is related to: IMPALA-4836 Kudu scans should operate within a memory constraint
• is related to: IMPALA-7297 Reject reservation increase if it would exceed soft limit
• relates to: IMPALA-7465 Hit memory limit in TestScanMemLimit.test_kudu_scan_mem_usage on ASAN build
• relates to: IMPALA-11068 Query hit OOM under high decompression activity
• relates to: IMPALA-7296 Soft limit for memory queue in scan node row batch queue