[SPARK-3434] Distributed block matrix

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Component/s: MLlib
• Labels: bulk-closed
• Assignee: Shivaram Venkataraman
• Reporter: Xiangrui Meng
• Created: 08/Sep/14 07:13
• Updated: 21/May/19 04:12
• Resolved: 21/May/19 04:12

Description

This JIRA is for discussing distributed matrices stored in block sub-matrices. The main challenge is the partitioning scheme to allow adding linear algebra operations in the future, e.g.:

1. matrix multiplication
2. matrix factorization (QR, LU, ...)

Let's discuss the partitioning and storage and how they fit into the above use cases.

Questions:

1. Should it be backed by a single RDD that contains all of the sub-matrices or many RDDs with each contains only one sub-matrix?

Issue Links

• depends upon: SPARK-3974 Block matrix abstracitons and partitioners
• depends upon: SPARK-3975 Block Matrix addition and multiplication
• depends upon: SPARK-3976 Detect block matrix partitioning schemes
• depends upon: SPARK-3977 Conversions between {Row, Coordinate}Matrix <-> BlockMatrix
• is depended upon by: SPARK-3435 Distributed matrix multiplication