[DRILL-6845] Eliminate duplicates for Semi Hash Join

• Type: Improvement
• Status: Reviewable
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 1.14.0
• Component/s: Execution - Relational Operators
• Assignee: Boaz Ben-Zvi
• Reporter: Boaz Ben-Zvi
• Reviewer: Timothy Farkas
• Created: 13/Nov/18 02:04
• Updated: 05/Nov/19 03:55

Description

Following DRILL-6735: The performance of the new Semi Hash Join may degrade if the build side contains excessive number of join-key duplicate rows; this mainly a result of the need to store all those rows first, before the hash table is built.

  Proposed solution: For Semi, the Hash Agg would create a Hash-Table initially, and use it to eliminate key-duplicate rows as they arrive.

  Proposed extra: That Hash-Table has an added cost (e.g. resizing). So perform "runtime stats" – Check initial number of incoming rows (e.g. 32k), and if the number of duplicates is less than some threshold (e.g. %20) – cancel that "early" hash table.

 

Issue Links

• is a child of: DRILL-6735 Enhance the Hash-Join Operator to perform Semi-Join