[SPARK-1485] Implement AllReduce

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Won't Fix
• Component/s: MLlib
• Assignee: Xiangrui Meng
• Reporter: Xiangrui Meng
• Created: 14/Apr/14 07:12
• Updated: 04/Jun/18 19:54
• Resolved: 10/Jul/14 01:23

Description

The current implementations of machine learning algorithms rely on the driver for some computation and data broadcasting. This will create a bottleneck at the driver for both computation and communication, especially in multi-model training. An efficient implementation of AllReduce (or AllAggregate) can help free the driver:

allReduce(RDD[T], (T, T) => T): RDD[T]

This JIRA is created for discussing how to implement AllReduce efficiently and possible alternatives.

Issue Links

• is related to: SPARK-24374 SPIP: Support Barrier Execution Mode in Apache Spark
• is related to: SPARK-2174 Implement treeReduce and treeAggregate
• links to: [Github] Pull Request #506 (mengxr)