[SPARK-23755] [Performance] Distinct elimination

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Ioana Delaney
• Created: 20/Mar/18 19:12
• Updated: 29/Mar/20 02:37

Description

The Distinct requirement can be removed if it is proved that the operation produces unique output.

select distinct d.deptno /* PK */, e.empname
from employee e, department d
where e.empno = d.mgrno /*PK = FK*/

Internal query after rewrite:

select d.deptno, e.empname
from employee e, department d
where e.empno = d.mgrno