[SPARK-16566] Bug in SparseMatrix multiplication with SparseVector

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 1.6.2
• Component/s: MLlib
• Assignee: Unassigned
• Reporter: Wilson
• Created: 15/Jul/16 08:31
• Updated: 22/Aug/16 23:22
• Resolved: 22/Aug/16 23:22

Description

In the org.apache.spark.mllib.linalg.BLAS.scala, the multiplication between SparseMatrix (sm) and SparseVector (sv) when sm is not transposed assume that the indices is sorted, but there is no validation to make sure that is the case, making the result returned wrongly.

This can be replicated simply by using spark-shell and entering these commands:

import org.apache.spark.mllib.linalg.SparseMatrix
import org.apache.spark.mllib.linalg.SparseVector
import org.apache.spark.mllib.linalg.DenseVector
import scala.collection.mutable.ArrayBuffer

val vectorIndices = Array(3,2)
val vectorValues = Array(0.1,0.2)
val size = 4

val sm = new SparseMatrix(size, size, Array(0, 0, 0, 1, 1), Array(0), Array(1.0))
val dm = sm.toDense
val sv = new SparseVector(size, vectorIndices, vectorValues)
val dv = new DenseVector(s.toArray)

sm.multiply(dv) == sm.multiply(sv)

sm.multiply(dv)
sm.multiply(sv)

Issue Links

• duplicates: SPARK-14707 Linear algebra: clarify light vs heavy constructors and accessors
• links to: [Github] Pull Request #14219 (wilson-lauw)