[SEDONA-64] Broadcast dedupParams in join judgements to reduce task deserialization time

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.1.0
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: Martin Andersson
• Created: 21/Sep/21 10:33
• Updated: 09/Oct/21 22:01
• Resolved: 09/Oct/21 22:01

Description

Join judgements requires large variables (dedupParams) that Spark will have to deserialize for every task.

For very large partition counts this adds a significant overhead. For jobs with sedona.join.numpartition=20000 (34k actual tasks) task deserialization time takes on average 200 ms.

By broadcasting dedupParams we've been able to reduce task deserialization time to 11 ms. Total job execution time is reduces by 20%.

https://spark.apache.org/docs/latest/rdd-programming-guide.html#broadcast-variables

Issue Links

• links to: GitHub Pull Request #545