[SPARK-3157] Avoid duplicated stats in DecisionTree extractLeftRightNodeAggregates

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 1.2.0
• Component/s: MLlib
• Assignee: Joseph K. Bradley
• Reporter: Joseph K. Bradley
• Created: 20/Aug/14 22:19
• Updated: 21/Jul/15 17:43
• Resolved: 21/Jul/15 17:43

Description

Improvement: computation, memory usage

For ordered features, extractLeftRightNodeAggregates() computes pairs of cumulative sums. However, these sums are redundant since they are simply cumulative sums accumulating from the left and right ends, respectively. Only compute one sum.
For unordered features, the left and right aggregates are essentially the same data, copied from the original aggregates, but shifted by one index. Avoid copying data.

Issue Links

• Is contained by: SPARK-3043 DecisionTree aggregation is inefficient