Description
InĀ SPARK-32290, we introduced several new types of HashedRelation
- EmptyHashedRelation
- EmptyHashedRelationWithAllNullKeys
They were all limited to used only in NAAJ scenario. These new HashedRelation could be applied to other scenario for performance improvements.
- EmptyHashedRelation could also be used in Normal AntiJoin for fast stop
- While AQE is on and buildSide is EmptyHashedRelationWithAllNullKeys, can convert NAAJ to a Empty LocalRelation to skip meaningless data iteration since in Single-Key NAAJ, if null key exists in BuildSide, will drop all records in streamedSide.
This Patch including two changes.
- using EmptyHashedRelation to do fast stop for common anti join as well
- In AQE, eliminate BroadcastHashJoin(NAAJ) if buildSide is a EmptyHashedRelationWithAllNullKeys