[SPARK-4285] Transpose RDD[Vector] to column store for ML

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Incomplete
• Component/s: MLlib
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Joseph K. Bradley
• Created: 07/Nov/14 00:48
• Updated: 08/Oct/19 05:41
• Resolved: 08/Oct/19 05:41

Description

For certain ML algorithms, a column store is more efficient than a row store (which is currently used everywhere). E.g., deep decision trees can be faster to train when partitioning by features.

Proposal: Provide a method with the following API (probably in util/):
```
def rowToColumnStore(data: RDD[Vector]): RDD[(Int, Vector)]
```
The input Vectors will be data rows/instances, and the output Vectors will be columns/features paired with column/feature indices.

*Question*: Is it important to maintain matrix structure? That is, should output Vectors in the same partition be adjacent columns in the matrix?

Issue Links

• blocks: SPARK-3717 DecisionTree, RandomForest: Partition by feature