[HIVE-11726] Pushed IN predicates to the metastore

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 2.0.0
• Assignee: Jesus Camacho Rodriguez
• Reporter: Jesus Camacho Rodriguez
• Created: 03/Sep/15 16:50
• Updated: 15/Nov/20 14:41
• Resolved: 06/Nov/15 17:05

Description

The PointLookupOptimizer can turn off some of the optimizations due to its use of tuple IN() clauses.

HIVE-11573 introduced the extraction of sub-clauses that could be pushed down till the TableScan operators, though they wouldn't be pushed down to the metastore.

In this issue, we tackle this problem by extending the filter parser of the metastore to support IN clauses, including multiple columns. This allows to push those additional predicates down throw directSQL to the metastore.

Issue Links

• is blocked by: HIVE-11634 Support partition pruning for IN(STRUCT(partcol, nonpartcol..)...)
• relates to: SPARK-33416 Avoid Hive metastore stack overflow when InSet predicate have many values
• relates to: HIVE-11573 PointLookupOptimizer can be pessimistic at a low nDV
• links to: RB link