[IMPALA-7297] Reject reservation increase if it would exceed soft limit

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Impala 3.1.0
• Component/s: Backend
• Labels: resource-management
• Target Version: Impala 3.1.0
• Epic Color: ghx-label-4
• Assignee: Tim Armstrong
• Reporter: Tim Armstrong
• Created: 13/Jul/18 17:50
• Updated: 14/Aug/18 22:52
• Resolved: 02/Aug/18 15:37

Description

Currently the MemTracker only has a hard limit, which is applied to any memory increase.

One problem we have with the interaction between reserved and non-reserved memory is that the reserved memory is capped to grow up to 80% of the query memory limit unless it hits the query memory limit. If non-reserved memory grows to > 20% of the query memory limit and reserved memory grows until it hits the hard limit, then we almost always end up with a query failure because there's no headroom for additional non-reserved allocations.

We could reduce query failures if reservation increases were rejected earlier to preserve some headroom for non-reserved allocations above 20% of the query memory limit.

The proposal is that reservation increases (aside from the initial reservation) should be rejected if they exceed (x * limit). We could determine x with some experimentation, but 5-10% might be a good value.

Issue Links

• relates to: IMPALA-7096 Ensure no memory limit exceeded regressions from IMPALA-4835 because of non-reserved memory