[SPARK-2372] Grouped Optimization/Learning

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 1.0.1, 1.0.2, 1.1.0
• Component/s: MLlib
• Assignee: Unassigned
• Reporter: Kyle Ellrott
• Created: 05/Jul/14 03:16
• Updated: 07/Nov/16 18:03
• Resolved: 01/Mar/15 11:17

Description

The purpose of this patch is the enable MLLib to better handle scenarios where the user would want to do learning on multiple feature/label sets at the same time. Rather then schedule each learning task separately, this patch lets the user create a single RDD with an Int key to represent the 'group' sets of entries belong to.

This patch establishing the GroupedOptimizer trait, for which GroupedGradientDescent has been implemented. This systems differs from the original Optimizer trait in that the original optimize method accepted RDD[(Int, Vector)] the new GroupedOptimizer accepts RDD[(Int, (Double, Vector))].
The difference is that the GroupedOptimizer uses a 'group' ID key in the RDD to multiplex multiple optimization operations in the same RDD.

This patch also establishes the GroupedGeneralizedLinearAlgorithm trait, for which the 'run' method has had the RDD[LabeledPoint] input replaced with RDD[(Int,LabeledPoint)].

This patch also provides a unit test and utility to take the results of MLUtils.kFold and turn it into a single grouped RDD, ready for simultaneous learning.

https://github.com/apache/spark/pull/1292

Issue Links

• relates to: SPARK-5256 Improving MLlib optimization APIs
• relates to: SPARK-18303 CLONE - Improving MLlib optimization APIs
• links to: [Github] Pull Request #1292 (kellrott)