[SPARK-14322] Use treeAggregate instead of reduce in OnlineLDAOptimizer

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.3.1, 1.4.1, 1.5.2, 1.6.1, 2.0.0
• Fix Version/s: 1.5.3, 1.6.2, 2.0.0
• Component/s: ML, MLlib
• Target Version/s: 1.5.3, 1.6.2, 2.0.0
• Assignee: yuhao yang
• Reporter: Joseph K. Bradley
• Shepherd: Joseph K. Bradley
• Created: 31/Mar/16 23:25
• Updated: 06/Apr/16 18:39
• Resolved: 06/Apr/16 18:39

Description

OnlineLDAOptimizer uses RDD.reduce in two places where it could use treeAggregate. This can cause scalability issues. This should be an easy fix.

This is also a bug since it modifies the first argument to reduce, so we should use aggregate or treeAggregate.

See this line: https://github.com/apache/spark/blob/f12f11e578169b47e3f8b18b299948c0670ba585/mllib/src/main/scala/org/apache/spark/mllib/clustering/LDAOptimizer.scala#L452
and a few lines below it.

Issue Links

• is required by: SPARK-5572 LDA improvement listing
• links to: [Github] Pull Request #12106 (hhbyyh)