[SPARK-17792] L-BFGS solver for linear regression does not accept general numeric label column types

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 2.0.2, 2.1.0
• Component/s: ML
• Assignee: Seth Hendrickson
• Reporter: Seth Hendrickson
• Created: 05/Oct/16 23:12
• Updated: 07/Oct/16 04:18
• Resolved: 07/Oct/16 04:13

Description

There's a bug in accepting numeric types for linear regression. We cast the label to DoubleType in one spot where we use normal solver, but not for the l-bfgs solver. The following can reproduce the problem:

import org.apache.spark.ml.feature.LabeledPoint
import org.apache.spark.ml.linalg.{Vector, DenseVector, Vectors}
import org.apache.spark.ml.regression.LinearRegression
import org.apache.spark.sql.types._

val df = Seq(LabeledPoint(1.0, Vectors.dense(1.0))).toDF().withColumn("weight", lit(1.0).cast(LongType))
val lr = new LinearRegression().setSolver("l-bfgs").setWeightCol("weight")
lr.fit(df)

Issue Links

• links to: [Github] Pull Request #15364 (sethah)