[IMPALA-5013] Re-evaluate our approach to per-operator memory estimates

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Done
• Affects Version/s: Impala 2.8.0
• Fix Version/s: Not Applicable
• Component/s: Frontend
• Labels: resource-management
• Target Version: Product Backlog
• Assignee: Unassigned
• Reporter: Tim Armstrong
• Created: 02/Mar/17 19:55
• Updated: 05/Dec/18 23:34
• Resolved: 05/Dec/18 23:34

Description

The way that memory estimates are computed for PlanNodes and Sinks are ad-hoc and in some cases much less accurate than they could be. We should clarify what the memory estimates mean, how they should be computed and then systematically fix them.

In general it's difficult to produce accurate memory estimates, because it depends on having accurate estimates of cardinality and other runtime parameters, so this JIRA isn't meant to guarantee any specific level of accuracy of estimates, just to generally improve the estimates and clarify what they mean and how they should be calculated

We should also consider deprecating or removing these estimates, unless they are useful for computing "ideal" memory in IMPALA-3706.

Issue Links

• relates to: IMPALA-2945 Pre-aggregation cardinality estimates do not take into account data distribution