Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-12372

Document limitations of MLlib local linear algebra

    XMLWordPrintableJSON

Details

    • Documentation
    • Status: Resolved
    • Major
    • Resolution: Incomplete
    • 1.5.2
    • None
    • Documentation, MLlib

    Description

      This JIRA is now for documenting limitations of MLlib's local linear algebra types. Basically, we should make it clear in the user guide that they provide simple functionality but are not a full-fledged local linear library. We should also recommend libraries for users to use in the meantime: probably Breeze for Scala (and Java?) and numpy/scipy for Python.

      Original JIRA title: Unary operator "-" fails for MLlib vectors

      Original JIRA text, as an example of the need for better docs:
      Consider the following snippet in pyspark 1.5.2:

      >>> from pyspark.mllib.linalg import Vectors
      >>> x = Vectors.dense([0.0, 1.0, 0.0, 7.0, 0.0])
      >>> x
      DenseVector([0.0, 1.0, 0.0, 7.0, 0.0])
      >>> -x
      Traceback (most recent call last):
        File "<stdin>", line 1, in <module>
      TypeError: func() takes exactly 2 arguments (1 given)
      >>> y = Vectors.dense([2.0, 0.0, 3.0, 4.0, 5.0])
      >>> y
      DenseVector([2.0, 0.0, 3.0, 4.0, 5.0])
      >>> x-y
      DenseVector([-2.0, 1.0, -3.0, 3.0, -5.0])
      >>> -y+x
      Traceback (most recent call last):
        File "<stdin>", line 1, in <module>
      TypeError: func() takes exactly 2 arguments (1 given)
      >>> -1*x
      DenseVector([-0.0, -1.0, -0.0, -7.0, -0.0])
      

      Clearly, the unary operator - (minus) for vectors fails, giving errors for expressions like -x and -y+x, despite the fact that x-y behaves as expected.
      The last operation, -1*x, although mathematically "correct", includes minus signs for the zero entries, which again is normally not expected.

      Attachments

        Activity

          People

            Unassigned Unassigned
            ctsats Christos Iraklis Tsatsoulis
            Votes:
            1 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: