[HIVE-15192] Use Calcite to de-correlate and plan subqueries

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.2.0
• Component/s: Logical Optimizer
• Labels: sub-query
• Assignee: Vineet Garg
• Reporter: Vineet Garg
• Created: 14/Nov/16 02:27
• Updated: 26/Jul/17 00:03
• Resolved: 16/Dec/16 18:28

Description

HIVE currently tranform subqueries into SEMI-JOIN or LEFT OUTER JOIN. This transformation occurs on query AST before generating logical plan. These transformations are described at Link to original spec . Such transformations aren't able to handle a lot of subqueries, as a result HIVE imposes various restrictions on the type of queries it could handle e.g. Hive disallows nested subqueries. All current restrictions are detailed in above linked document.

This patch is 1st phase of getting rid of these transformations and leverage Calcite's functionality to plan such queries.
Next phases will be lifting restrictions one by one.

Note that this patch already lifts one restriction Restriction.6.m (The LHS in a SubQuery must have all its Column References be qualified)

Known issues with this patch are:

• Return path tests fails for various reasons and are currently disabled. We plan to fix and re-enable this later.
• Semi-join optimization (HIVE-15227) is disabled by default as it doesn't work with this patch. We plan to fix this and re-enable it by default.

Issue Links

• links to: ReviewBoard