[SPARK-14707] Linear algebra: clarify light vs heavy constructors and accessors

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Component/s: ML
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Joseph K. Bradley
• Created: 18/Apr/16 18:01
• Updated: 21/May/19 04:33
• Resolved: 21/May/19 04:33

Description

MLlib linear algebra provides methods for constructing Vectors and Matrices and for accessing the vector/matrix data. There are currently 2 types of these constructors and accessors:

• light: avoid data copy and validation, useful for converting between MLlib types and numpy, Breeze, etc.
• heavy: copy or validate data, useful for constructing MLlib types from user inputs

MLlib is not very consistent about these and does not document which ops are light vs. heavy. This JIRA is for:

• First, discussing which ops should be light vs heavy to choose a consistent API
• Next, creating subtasks for Scala and Python for updating the implementations and clarifying the docs

Issue Links

• is blocked by: SPARK-13944 Separate out local linear algebra as a standalone module without Spark dependency
• is duplicated by: SPARK-16566 Bug in SparseMatrix multiplication with SparseVector
• supercedes: SPARK-14697 mllib DenseMatrix toArray could use the internal values