[IMPALA-8483] Make coordinator fragment lighter-weight

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Frontend
• Labels: resource-management scalability
• Epic Color: ghx-label-1
• Assignee: Bikramjeet Vig
• Reporter: Tim Armstrong
• Created: 02/May/19 18:07
• Updated: 29/Dec/20 21:34

Description

it's possible to write queries with an arbitrarily expensive coordinator fragment. One example is a query with a set of unpartitioned analytic functions each with a different ORDER BY, which results in many SORT nodes in the coordinator fragment.

This is a problem for dedicated coordinators because it makes the resource consumption unpredictable.

It would be useful if we could offload that work to executors and guarantee that only "lightweight' operators are part of the coordinator fragment, e.g. operators that take O(#rows returned * log(#executors)) time and O(#executors) memory - regular exchanges, merging exchanges, TOP-N, non-grouping aggregation, etc.

I think this can be done in general by inserting an additional exchange on top of the "expensive" part of a coordinator fragment, and then ensuring that that fragment gets scheduled on an executor.

Issue Links

• is related to: IMPALA-7486 Admit less memory on dedicated coordinator for admission control purposes
• relates to: IMPALA-9850 Avoid doing expensive join at the coordinator