[DRILL-4667] Improve memory footprint of broadcast joins

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.6.0
• Component/s: Execution - Relational Operators
• Assignee: Boaz Ben-Zvi
• Reporter: Aman Sinha
• Created: 11/May/16 16:04
• Updated: 23/Mar/20 09:10

Description

For broadcast joins, currently Drill optimizes the data transfer across the network for broadcast table by sending a single copy to the receiving node which then distributes it to all minor fragments running on that particular node. However, each minor fragment builds its own hash table (for a hash join) using this broadcast table. We can substantially improve the memory footprint by having a shared copy of the hash table among multiple minor fragments on a node.