[SPARK-20687] mllib.Matrices.fromBreeze may crash when converting from Breeze sparse matrix

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.1.1
• Fix Version/s: 2.1.2, 2.2.0
• Component/s: MLlib
• Flags: Patch
• Assignee: Ignacio Bermudez Corrales
• Reporter: Ignacio Bermudez Corrales
• Created: 10/May/17 04:24
• Updated: 22/May/17 09:29
• Resolved: 22/May/17 09:28

Description

Conversion of Breeze sparse matrices to Matrix is broken when matrices are product of certain operations. This problem I think is caused by the update method in Breeze CSCMatrix when they add provisional zeros to the data for efficiency.

This bug is serious and may affect at least BlockMatrix addition and substraction

http://stackoverflow.com/questions/33528555/error-thrown-when-using-blockmatrix-add/43883458#43883458

The following code, reproduces the bug (Check test("breeze conversion bug"))

https://github.com/ghoto/spark/blob/test-bug/CSCMatrixBreeze/mllib/src/test/scala/org/apache/spark/mllib/linalg/MatricesSuite.scala

MatricesSuite.scala

  test("breeze conversion bug") {
    // (2, 0, 0)
    // (2, 0, 0)
    val mat1Brz = Matrices.sparse(2, 3, Array(0, 2, 2, 2), Array(0, 1), Array(2, 2)).asBreeze
    // (2, 1E-15, 1E-15)
    // (2, 1E-15, 1E-15
    val mat2Brz = Matrices.sparse(2, 3, Array(0, 2, 4, 6), Array(0, 0, 0, 1, 1, 1), Array(2, 1E-15, 1E-15, 2, 1E-15, 1E-15)).asBreeze
    // The following shouldn't break
    val t01 = mat1Brz - mat1Brz
    val t02 = mat2Brz - mat2Brz
    val t02Brz = Matrices.fromBreeze(t02)
    val t01Brz = Matrices.fromBreeze(t01)

    val t1Brz = mat1Brz - mat2Brz
    val t2Brz = mat2Brz - mat1Brz
    // The following ones should break
    val t1 = Matrices.fromBreeze(t1Brz)
    val t2 = Matrices.fromBreeze(t2Brz)

  }

Issue Links

• Is contained by: SPARK-11507 Error thrown when using BlockMatrix.add
• links to: [Github] Pull Request #17940 (ghoto)