[SPARK-38588] Validate input dataset of ml.classification

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Resolved
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: ML
• Assignee: Unassigned
• Reporter: Ruifeng Zheng
• Created: 17/Mar/22 10:29
• Updated: 02/Apr/22 14:42
• Resolved: 24/Mar/22 08:05

Description

LinearSVC should fail fast if the input dataset contains invalid values.

 

import org.apache.spark.ml.feature._
import org.apache.spark.ml.linalg._
import org.apache.spark.ml.classification._
import org.apache.spark.ml.clustering._
val df = sc.parallelize(Seq(LabeledPoint(1.0, Vectors.dense(1.0, Double.NaN)), LabeledPoint(0.0, Vectors.dense(Double.PositiveInfinity, 2.0)))).toDF()

val svc = new LinearSVC()
val model = svc.fit(df)

scala> model.intercept
res0: Double = NaN

scala> model.coefficients
res1: org.apache.spark.ml.linalg.Vector = [NaN,NaN] 

Issue Links

• links to: [Github] Pull Request #35893 (zhengruifeng)
• links to: [Github] Pull Request #35893 (zhengruifeng)
• links to: [Github] Pull Request #36026 (jackylee-ch)