[SPARK-21285] VectorAssembler should report the column name when data type used is not supported

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.1.1
• Fix Version/s: 2.3.0
• Component/s: ML, MLlib
• Assignee: Yan Facai (颜发才)
• Reporter: Jacek Laskowski
• Created: 03/Jul/17 08:40
• Updated: 07/Jul/17 10:35
• Resolved: 07/Jul/17 10:35

Description

Found while answering Why does LogisticRegression fail with “IllegalArgumentException: org.apache.spark.ml.linalg.VectorUDT@3bfc3ba7”? on StackOverflow.

When VectorAssembler is configured to use columns of unsupported type only the type is printed out without the column name(s).

The column name(s) should be included too.

// label is of StringType type
val va = new VectorAssembler().setInputCols(Array("bc", "pmi", "label"))
scala> va.transform(training)
java.lang.IllegalArgumentException: Data type StringType is not supported.
  at org.apache.spark.ml.feature.VectorAssembler$$anonfun$transformSchema$1.apply(VectorAssembler.scala:121)
  at org.apache.spark.ml.feature.VectorAssembler$$anonfun$transformSchema$1.apply(VectorAssembler.scala:117)
  at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
  at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
  at org.apache.spark.ml.feature.VectorAssembler.transformSchema(VectorAssembler.scala:117)
  at org.apache.spark.ml.PipelineStage.transformSchema(Pipeline.scala:74)
  at org.apache.spark.ml.feature.VectorAssembler.transform(VectorAssembler.scala:54)
  ... 48 elided

Issue Links

• links to: [Github] Pull Request #18523 (facaiy)