[SPARK-12660] Rewrite except using anti-join

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Component/s: SQL
• Target Version/s: 2.0.0
• Assignee: Xiao Li
• Reporter: Reynold Xin
• Created: 05/Jan/16 22:36
• Updated: 29/Apr/16 07:32
• Resolved: 29/Apr/16 07:32

Description

Similar to SPARK-12656, we can rewrite except in the logical level using anti-join. This way, we can take advantage of all the benefits of join implementations (e.g. managed memory, code generation, broadcast joins).

Issue Links

• is blocked by: SPARK-12610 Add Anti Join Operators
• is blocked by: SPARK-14950 Anti Join is broken when using BroadcastHashJoin's unique key path
• is related to: SPARK-12656 Rewrite Intersect phyiscal plan using semi-join
• relates to: SPARK-13136 Data exchange (shuffle, broadcast) should only be handled by the exchange operator
• supercedes: SPARK-12709 Create local Except operator
• links to: [Github] Pull Request #12705 (lresende)
• links to: [Github] Pull Request #12736 (gatorsmile)