Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-12372

Document limitations of MLlib local linear algebra

    Details

    • Type: Documentation
    • Status: Resolved
    • Priority: Major
    • Resolution: Incomplete
    • Affects Version/s: 1.5.2
    • Fix Version/s: None
    • Component/s: Documentation, MLlib
    • Labels:

      Description

      This JIRA is now for documenting limitations of MLlib's local linear algebra types. Basically, we should make it clear in the user guide that they provide simple functionality but are not a full-fledged local linear library. We should also recommend libraries for users to use in the meantime: probably Breeze for Scala (and Java?) and numpy/scipy for Python.

      Original JIRA title: Unary operator "-" fails for MLlib vectors

      Original JIRA text, as an example of the need for better docs:
      Consider the following snippet in pyspark 1.5.2:

      >>> from pyspark.mllib.linalg import Vectors
      >>> x = Vectors.dense([0.0, 1.0, 0.0, 7.0, 0.0])
      >>> x
      DenseVector([0.0, 1.0, 0.0, 7.0, 0.0])
      >>> -x
      Traceback (most recent call last):
        File "<stdin>", line 1, in <module>
      TypeError: func() takes exactly 2 arguments (1 given)
      >>> y = Vectors.dense([2.0, 0.0, 3.0, 4.0, 5.0])
      >>> y
      DenseVector([2.0, 0.0, 3.0, 4.0, 5.0])
      >>> x-y
      DenseVector([-2.0, 1.0, -3.0, 3.0, -5.0])
      >>> -y+x
      Traceback (most recent call last):
        File "<stdin>", line 1, in <module>
      TypeError: func() takes exactly 2 arguments (1 given)
      >>> -1*x
      DenseVector([-0.0, -1.0, -0.0, -7.0, -0.0])
      

      Clearly, the unary operator - (minus) for vectors fails, giving errors for expressions like -x and -y+x, despite the fact that x-y behaves as expected.
      The last operation, -1*x, although mathematically "correct", includes minus signs for the zero entries, which again is normally not expected.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              ctsats Christos Iraklis Tsatsoulis
            • Votes:
              1 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: