[SPARK-12811] Estimator interface for generalized linear models (GLMs)

• Type: New Feature
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 2.0.0
• Component/s: ML
• Target Version/s: 2.0.0
• Assignee: Yanbo Liang
• Reporter: Xiangrui Meng
• Shepherd: Xiangrui Meng
• Created: 13/Jan/16 19:22
• Updated: 01/Mar/16 16:52
• Resolved: 01/Mar/16 16:48

Description

In Spark 1.6, MLlib provides logistic regression and linear regression with L1/L2/elastic-net regularization. We want to expand the support of generalized linear models (GLMs) in 2.0, e.g., Poisson/Gamma families and more link functions. SPARK-9835 implements a GLM solver for the case when the number of features is small. We also need to design an interface for GLMs.

In SparkR, we can simply follow glm or glmnet. On the Python/Scala/Java side, the interface should be consistent with LinearRegression and LogisticRegression, e.g.,

val glm = new GeneralizedLinearModel()
  .setFamily("poisson")
  .setSolver("irls")

It would be great if LinearRegression and LogisticRegression can reuse code from GeneralizedLinearModel.

Issue Links

• is depended upon by: SPARK-13597 Python API for GeneralizedLinearRegression
• is related to: SPARK-12566 GLM model family, link function support in SparkR:::glm
• relates to: SPARK-9835 Iteratively reweighted least squares solver for GLMs
• links to: [Github] Pull Request #11136 (yanboliang)