[SPARK-12372] Document limitations of MLlib local linear algebra

• Type: Documentation
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 1.5.2
• Component/s: Documentation, MLlib
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Christos Iraklis Tsatsoulis
• Created: 16/Dec/15 18:22
• Updated: 21/May/19 04:33
• Resolved: 21/May/19 04:33

Description

This JIRA is now for documenting limitations of MLlib's local linear algebra types. Basically, we should make it clear in the user guide that they provide simple functionality but are not a full-fledged local linear library. We should also recommend libraries for users to use in the meantime: probably Breeze for Scala (and Java?) and numpy/scipy for Python.

Original JIRA title: Unary operator "-" fails for MLlib vectors

Original JIRA text, as an example of the need for better docs:
Consider the following snippet in pyspark 1.5.2:

>>> from pyspark.mllib.linalg import Vectors
>>> x = Vectors.dense([0.0, 1.0, 0.0, 7.0, 0.0])
>>> x
DenseVector([0.0, 1.0, 0.0, 7.0, 0.0])
>>> -x
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: func() takes exactly 2 arguments (1 given)
>>> y = Vectors.dense([2.0, 0.0, 3.0, 4.0, 5.0])
>>> y
DenseVector([2.0, 0.0, 3.0, 4.0, 5.0])
>>> x-y
DenseVector([-2.0, 1.0, -3.0, 3.0, -5.0])
>>> -y+x
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: func() takes exactly 2 arguments (1 given)
>>> -1*x
DenseVector([-0.0, -1.0, -0.0, -7.0, -0.0])

Clearly, the unary operator - (minus) for vectors fails, giving errors for expressions like -x and -y+x, despite the fact that x-y behaves as expected.
The last operation, -1*x, although mathematically "correct", includes minus signs for the zero entries, which again is normally not expected.