[SPARK-6321] Adapt the number of partitions used by the Exchange rule to the cluster specifications

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 1.3.1
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Óscar Puertas
• Created: 13/Mar/15 13:39
• Updated: 15/Sep/15 23:04
• Resolved: 15/Sep/15 23:04

Description

Currently, the exchange rule is using a default value of 200 fixed partitions. In my opinion, it would be nice if we can set that default value to something more related to the cluster specifications instead of a magic number.

My proposal is to use the default.parallelism value in the Spark configuration.

Issue Links

• relates to: SPARK-6377 Set the number of map output partitions for Exchange operator automatically based on the size of input tables and the reduce-side operation.
• links to: [Github] Pull Request #5013 (ojkar)