[HIVE-8649] Increase level of parallelism in reduce phase [Spark Branch]

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.1.0
• Component/s: Spark
• Assignee: Jimmy Xiang
• Reporter: Brock Noland
• Created: 29/Oct/14 18:19
• Updated: 29/May/15 02:31
• Resolved: 06/Nov/14 23:06

Description

We calculate the number of reducers based on the same code for MapReduce. However, reducers are vastly cheaper in Spark and it's generally recommended we have many more reducers than in MR.

Sandy Ryza who works on Spark has some ideas about a heuristic.