[SPARK-32573] Anti Join Improvement with EmptyHashedRelation and EmptyHashedRelationWithAllNullKeys - ASF JIRA

Attach files

Attach Screenshot

Voters

Watch issue

Watchers

Create sub-task

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

In ~~SPARK-32290~~, we introduced several new types of HashedRelation

They were all limited to used only in NAAJ scenario. These new HashedRelation could be applied to other scenario for performance improvements.

EmptyHashedRelation could also be used in Normal AntiJoin for fast stop
While AQE is on and buildSide is EmptyHashedRelationWithAllNullKeys, can convert NAAJ to a Empty LocalRelation to skip meaningless data iteration since in Single-Key NAAJ, if null key exists in BuildSide, will drop all records in streamedSide.

This Patch including two changes.

using EmptyHashedRelation to do fast stop for common anti join as well
In AQE, eliminate BroadcastHashJoin(NAAJ) if buildSide is a EmptyHashedRelationWithAllNullKeys