[SPARK-32112] Easier way to repartition/coalesce DataFrames based on the number of parallel tasks that Spark can process at the same time

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.0.0
• Component/s: Spark Core
• Assignee: Unassigned
• Reporter: Noritaka Sekiyama
• Created: 27/Jun/20 00:18
• Updated: 28/Jun/20 08:07

Description

Repartition/coalesce is very important to optimize Spark application's performance, however, a lot of users are struggling with determining the number of partitions.
This issue is to add a easier way to repartition/coalesce DataFrames based on the number of parallel tasks that Spark can process at the same time.

It will help Spark users to determine the optimal number of partitions.

Expected use-cases:

• repartition with the calculated parallel tasks

Notes:

• `SparkContext.maxNumConcurrentTasks` might help but it cannot be accessed by Spark apps.
• `SparkContext.getExecutorMemoryStatus` might help to calculate the number of available slots to process tasks.