[SPARK-6377] Set the number of map output partitions for Exchange operator automatically based on the size of input tables and the reduce-side operation.

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Component/s: SQL
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Yin Huai
• Created: 17/Mar/15 03:51
• Updated: 21/May/19 05:36
• Resolved: 21/May/19 05:36

Description

It will be helpful to automatically set the number of shuffle partitions based on the size of input tables and the operation at the reduce side for an Exchange operator.

Issue Links

• is related to: SPARK-6321 Adapt the number of partitions used by the Exchange rule to the cluster specifications
• is related to: SPARK-4630 Dynamically determine optimal number of partitions