[DRILL-6027] Implement spill to disk for the Hash Join

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.11.0
• Fix Version/s: 1.14.0
• Component/s: Execution - Relational Operators
• Labels: doc-complete
• Assignee: Boaz Ben-Zvi
• Reporter: Boaz Ben-Zvi
• Reviewer: Aman Sinha
• Created: 12/Dec/17 03:29
• Updated: 13/Mar/19 11:32
• Resolved: 17/May/18 22:05

Description

Implement the spill memory to disk (as needed) feature for the Hash Join operator (similar to the prior work on the Hash Aggregate).

A design draft document has been published:

https://docs.google.com/document/d/1-c_oGQY4E5d58qJYv_zc7ka834hSaB3wDQwqKcMoSAI/edit?usp=sharing

Functional spec is available: https://docs.google.com/document/d/1bPAddVCRxKHxi2RjqUvISIWXNqLdbRzZT9CWmanh4h0/edit

Issue Links

• causes: DRILL-7091 Query with EXISTS and correlated subquery fails with NPE in HashJoinMemoryCalculatorImpl$BuildSidePartitioningImpl
• causes: DRILL-6606 Hash Join returns incorrect data types when joining subqueries with limit 0
• links to: GitHub Pull Request #1248