[SPARK-48261] RoundRobin based coalesce in spark

• Type: New Feature
• Status: Resolved
• Priority: Minor
• Resolution: Abandoned
• Affects Version/s: 3.5.1
• Component/s: Spark Core
• Assignee: Unassigned
• Reporter: Subham Singhal
• Created: 13/May/24 11:26
• Updated: 14/May/24 04:10
• Resolved: 14/May/24 04:10

Description

Currently default coalsce does not take partition size into account and simply merges partitions. This often results in non-uniform data distribution. There have been proposal for size based coalesce(https://github.com/apache/spark/pull/27248).

I am proposing a custom roundrobin coalesce which will distribute data evenly across partitions within same executor.