[DRILL-6635] PartitionLimit for Lateral/Unnest

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.14.0
• Fix Version/s: 1.15.0
• Component/s: Execution - Relational Operators
• Labels: ready-to-commit
• Assignee: Sorabh Hamirwasia
• Reporter: Sorabh Hamirwasia
• Reviewer: Parth Chandra
• Created: 25/Jul/18 20:29
• Updated: 02/Aug/18 20:26
• Resolved: 02/Aug/18 20:26

Description

With batch processing changes in Lateral/Unnest the limit/TopN clause within Lateral-Unnest subquery will not work as expected since it will impose limit/TopN across RowId's. We need a new mechanism to apply these operators at rowId boundary.
For now we are planning to add support for only limit and hence need to have a new operator PartitionLimit which will get the partitionColumn on which the limit should be imposed. This will currently only support queries between lateral and unnest. 
For TopN we can still achieve that using combination of Sort and Partition Limit and later we can figure out how to address it directly within TopN or is it needed at all. Since the number of rows across EMIT boundary on which SORT will operate should not be big enough and mostly be done in memory.

Issue Links

• is required by: DRILL-5999 Add support for LATERAL join
• is required by: DRILL-6652 PartitionLimit changes for Lateral and Unnest