[SPARK-5997] Increase partition count without performing a shuffle

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Spark Core
• Assignee: Unassigned
• Reporter: Andrew Ash
• Created: 25/Feb/15 01:53
• Updated: 20/Dec/21 19:36

Description

When decreasing partition count with rdd.repartition() or rdd.coalesce(), the user has the ability to choose whether or not to perform a shuffle. However when increasing partition count there is no option of whether to perform a shuffle or not – a shuffle always occurs.

This Jira is to create a rdd.repartition(largeNum, shuffle=false) call that performs a repartition to a higher partition count without a shuffle.

The motivating use case is to decrease the size of an individual partition enough that the .toLocalIterator has significantly reduced memory pressure on the driver, as it loads a partition at a time into the driver.