[IMPALA-1728] sub-query with duplicate values used IN conditional operator should discard the duplicate values before applying the operator

• Type: New Feature
• Status: Resolved
• Priority: Minor
• Resolution: Duplicate
• Affects Version/s: Impala 2.0, Impala 2.1
• Component/s: Frontend
• Labels: performance planner tpc-ds
• Target Version: Product Backlog
• Assignee: Unassigned
• Reporter: Dileep Kumar
• Created: 02/Feb/15 21:49
• Updated: 15/Dec/20 19:57
• Resolved: 15/Dec/20 19:57

Description

When running the TPC-DS Q95 we found that it usages a result of CTE in IN conditional later in query.
In this case CTE generates too many duplicate values for the same column which is used in conditional. When applied the DISTINCT to CTE it took 40% less time to complete.
The timings(in Sec.) are as:
Without DISTINCT : 1240
With DISTINCT : 728

Both versions of the query are attached.

Issue Links

• duplicates: IMPALA-1270 Consider adding distinct aggregation to subqueries as perf optimization