[IMPALA-9416] Reduce minimum reservation for shared broadcast join builders

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Frontend
• Labels: multithreading resource-management
• Epic Link: Multi-threaded query execution - Further Enhancements
• Target Version: Product Backlog
• Epic Color: ghx-label-1
• Assignee: Unassigned
• Reporter: Tim Armstrong
• Created: 21/Feb/20 23:57
• Updated: 01/Jun/20 19:09

Description

This is an optimisation that I'm deferring from IMPALA-9156. The initial version of IMPALA-9156 will be conservative in the amount of memory it reserves for the broadcast join builder.

We should make it more precise to avoid blowing up the memory requirements. Specifically:

• the reservation should depend on the number of consuming fragments scheduled, not mt_dop
• we may want to look at how to reduce the amount of memory reserved per probe side. E.g. use smaller buffer sizes for the probe streams, under the assumption that a broadcast join is unlikely to spill.

Issue Links

• depends upon: IMPALA-9156 Share broadcast join builds between fragments