[IMPALA-9255] Avoid unnecessary concurrent resource consumption by fragments.

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Frontend
• Labels: multithreading
• Epic Link: Multi-threaded query execution - Further Enhancements
• Epic Color: ghx-label-12
• Assignee: Unassigned
• Reporter: Tim Armstrong
• Created: 16/Dec/19 21:48
• Updated: 01/Jun/20 19:08

Description

In Planner.computeResourceReqs(), we assume that all fragment's peak resource consumption occurs at the same time, resulting in higher than necessary estimates. There are a few things that need to change here to address this inefficiency:

• We should consider changing the backend logic so that fragments don't start up and claim resources until their dependencies are ready.
• We need to change the backend logic so that resources can be released when needed, e.g. the join build fragment's thread should exit once the build is done, instead of waiting for the probe threads to all exit
• We need to update the planner to reflect the new behaviour.