[IMPALA-10099] Push down DISTINCT aggregation for EXCEPT/INTERSECT

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Impala 4.0.0
• Epic Color: ghx-label-13
• Assignee: Shant Hovsepian
• Reporter: Shant Hovsepian
• Created: 22/Aug/20 03:38
• Updated: 22/Dec/20 02:16
• Resolved: 09/Sep/20 21:30

Description

The implementation of SetOperations for EXCEPT/INTERSECT in IMPALA-9943 produced query rewrites that would apply DISTINCT aggregation after exchanges for distributed plans. In case where the query can be directly rewritten to apply the DISTINCT to the set operation operands would result in better performance for most large queries.

This should help the performance TPC-DS Q14 which does an INTERSECT of queries with large result sets that contain many duplicates.

In general it would better to have DISTINCT move around optimization phase during planning which would handle this case as well as many others.

Issue Links

• is superceded by: IMPALA-5260 Have query optimizer make joined tables distinct to improve performance