[SPARK-18946] treeAggregate will be low effficiency when aggregate high dimension vectors in ML algorithm

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Component/s: ML, MLlib
• Labels: bulk-closed features
• Assignee: Unassigned
• Reporter: zunwen you
• Created: 20/Dec/16 13:12
• Updated: 25/May/21 01:49
• Resolved: 25/May/21 01:40

Description

In many machine learning algorithms, we have to treeAggregate large vectors/arrays due to the large number of features. Unfortunately, the treeAggregate operation of RDD will be low efficiency when the dimension of vectors/arrays is bigger than million. Because high dimension of vector/array always occupy more than 100MB Memory, transferring a 100MB element among executors is pretty low efficiency in Spark.

Issue Links

• links to: [Github] Pull Request #17000 (ZunwenYou)
• links to: GitHub Pull Request #17000