[SPARK-24941] Add RDDBarrier.coalesce() function

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.0
• Component/s: Spark Core
• Epic Link: Support Barrier Execution Mode
• Assignee: Unassigned
• Reporter: Xingbo Jiang
• Created: 27/Jul/18 03:17
• Updated: 25/Apr/24 19:22

Description

https://github.com/apache/spark/pull/21758#discussion_r204917245

The number of partitions from the input data can be unexpectedly large, eg. if you do

sc.textFile(...).barrier().mapPartitions()

The number of input partitions is based on the hdfs input splits. We shall provide a way in RDDBarrier to enable users to specify the number of tasks in a barrier stage. Maybe something like RDDBarrier.coalesce(numPartitions: Int) .