[SPARK-21723] Can't write LibSVM - key not found: numFeatures

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.2.0, 2.3.0
• Fix Version/s: 2.2.1, 2.3.0
• Component/s: Input/Output, ML
• Assignee: Jan Vršovský
• Reporter: Jan Vršovský
• Created: 14/Aug/17 09:44
• Updated: 16/Aug/17 07:25
• Resolved: 16/Aug/17 07:24

Description

Writing a dataset to LibSVM format raises an exception

java.util.NoSuchElementException: key not found: numFeatures

Happens only when the dataset was NOT read from a LibSVM format before (because otherwise numFeatures is in its metadata). Steps to reproduce:

import org.apache.spark.ml.linalg.Vectors
val rawData = Seq((1.0, Vectors.sparse(3, Seq((0, 2.0), (1, 3.0)))),
                  (4.0, Vectors.sparse(3, Seq((0, 5.0), (2, 6.0)))))
val dfTemp = spark.sparkContext.parallelize(rawData).toDF("label", "features")
dfTemp.coalesce(1).write.format("libsvm").save("...filename...")

PR with a fix and unit test is ready - see https://github.com/apache/spark/pull/18872.