[SPARK-2505] Weighted Regularizer

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Won't Fix
• Component/s: MLlib
• Assignee: Unassigned
• Reporter: DB Tsai
• Created: 16/Jul/14 02:50
• Updated: 07/Nov/16 18:03
• Resolved: 30/Jun/15 16:40

Description

The current implementation of regularization in linear model is using `Updater`, and this design has couple issues as the following.

1) It will penalize all the weights including intercept. In machine learning training process, typically, people don't penalize the intercept.
2) The `Updater` has the logic of adaptive step size for gradient decent, and we would like to clean it up by separating the logic of regularization out from updater to regularizer so in LBFGS optimizer, we don't need the trick for getting the loss and gradient of objective function.

In this work, a weighted regularizer will be implemented, and users can exclude the intercept or any weight from regularization by setting that term with zero weighted penalty. Since the regularizer will return a tuple of loss and gradient, the adaptive step size logic, and soft thresholding for L1 in Updater will be moved to SGD optimizer.

Issue Links

• relates to: SPARK-5256 Improving MLlib optimization APIs
• relates to: SPARK-18303 CLONE - Improving MLlib optimization APIs
• links to: [Github] Pull Request #1518 (dbtsai)