Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-5656

NegativeArraySizeException in EigenValueDecomposition.symmetricEigs for large n and/or large k

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • None
    • 1.4.0
    • MLlib
    • None

    Description

      Large values of n or k in EigenValueDecomposition.symmetricEigs will fail with a NegativeArraySizeException. Specifically, this occurs when 2*n*k > Integer.MAX_VALUE. These values are currently unchecked and allow for the array to be initialized to a value greater than Integer.MAX_VALUE. I have written the below 'require' to fail this condition gracefully. I will submit a pull request.

      require(ncv * n.toLong < Integer.MAX_VALUE, "Product of 2*k*n must be smaller than " +
      s"Integer.MAX_VALUE. Found required eigenvalues k = $k and matrix dimension n = $n")

      Here is the exception that occurs from computeSVD with large k and/or n:

      Exception in thread "main" java.lang.NegativeArraySizeException
      at org.apache.spark.mllib.linalg.EigenValueDecomposition$.symmetricEigs(EigenValueDecomposition.scala:85)
      at org.apache.spark.mllib.linalg.distributed.RowMatrix.computeSVD(RowMatrix.scala:258)
      at org.apache.spark.mllib.linalg.distributed.RowMatrix.computeSVD(RowMatrix.scala:190)

      Attachments

        Activity

          People

            mbittmann Mark Bittmann
            mbittmann Mark Bittmann
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: