[SPARK-9835] Iteratively reweighted least squares solver for GLMs

• Type: New Feature
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Component/s: ML, MLlib
• Target Version/s: 2.0.0
• Assignee: Yanbo Liang
• Reporter: Xiangrui Meng
• Shepherd: Xiangrui Meng
• Created: 11/Aug/15 19:25
• Updated: 28/Jan/16 22:29
• Resolved: 28/Jan/16 22:29

Description

After SPARK-9834, we can implement iteratively reweighted least squares (IRLS) solver for GLMs with other families and link functions. It could provide R-like summary statistics after training, but the number of features cannot be very large, e.g. more than 4096.

Issue Links

• blocks: SPARK-9837 Provide R-like summary statistics for GLMs via iteratively reweighted least squares
• depends upon: SPARK-9834 Normal equation solver for ordinary least squares
• is related to: SPARK-12811 Estimator interface for generalized linear models (GLMs)
• is related to: SPARK-9838 Support Poisson family in SparkR:::glm
• relates to: SPARK-9647 MLlib + SparkR integration for 1.6
• links to: [Github] Pull Request #10639 (yanboliang)