[SEDONA-217] Automatically broadcast small datasets

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.4.0
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: Martin Andersson
• Created: 12/Dec/22 12:30
• Updated: 13/Mar/23 21:38
• Resolved: 04/Jan/23 22:36

Description

Spark uses stats for each side of a join and automatically switches to a broadcast join if size is below threshold. Sedona could do the same.

This is where it's done in Spark.
https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala

Issue Links

• links to: GitHub Pull Request #730