[SPARK-32573] Anti Join Improvement with EmptyHashedRelation and EmptyHashedRelationWithAllNullKeys

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.1.0
• Component/s: SQL
• Assignee: Leanken.Lin
• Reporter: Leanken.Lin
• Created: 08/Aug/20 07:56
• Updated: 11/Aug/20 06:24
• Resolved: 11/Aug/20 06:24

Description

In SPARK-32290, we introduced several new types of HashedRelation

• EmptyHashedRelation
• EmptyHashedRelationWithAllNullKeys

They were all limited to used only in NAAJ scenario. These new HashedRelation could be applied to other scenario for performance improvements.

• EmptyHashedRelation could also be used in Normal AntiJoin for fast stop
• While AQE is on and buildSide is EmptyHashedRelationWithAllNullKeys, can convert NAAJ to a Empty LocalRelation to skip meaningless data iteration since in Single-Key NAAJ, if null key exists in BuildSide, will drop all records in streamedSide.

This Patch including two changes.

• using EmptyHashedRelation to do fast stop for common anti join as well
• In AQE, eliminate BroadcastHashJoin(NAAJ) if buildSide is a EmptyHashedRelationWithAllNullKeys

Issue Links

• links to: [Github] Pull Request #29389 (leanken)
• links to: [Github] Pull Request #29389 (leanken)