[SPARK-6567] Large linear model parallelism via a join and reduceByKey

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Component/s: ML, MLlib
• Assignee: Unassigned
• Reporter: Reza Zadeh
• Shepherd: Xiangrui Meng
• Created: 27/Mar/15 08:27
• Updated: 27/Mar/17 00:01
• Resolved: 28/Feb/17 10:32

Description

To train a linear model, each training point in the training set needs its dot product computed against the model, per iteration. If the model is large (too large to fit in memory on a single machine) then SPARK-4590 proposes using parameter server.

There is an easier way to achieve this without parameter servers. In particular, if the data is held as a BlockMatrix and the model as an RDD, then each block can be joined with the relevant part of the model, followed by a reduceByKey to compute the dot products.

This obviates the need for a parameter server, at least for linear models. However, it's unclear how it compares performance-wise to parameter servers.

Issue Links

• is superceded by: SPARK-10078 Vector-free L-BFGS
• relates to: SPARK-6932 A Prototype of Parameter Server
• relates to: SPARK-4590 Early investigation of parameter server