[SPARK-16719] RandomForest: communicate fewer trees on each iteration

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 2.1.0
• Component/s: ML
• Target Version/s: 2.1.0
• Assignee: Joseph K. Bradley
• Reporter: Joseph K. Bradley
• Created: 25/Jul/16 21:11
• Updated: 23/Sep/16 05:28
• Resolved: 23/Sep/16 05:28

Description

RandomForest currently sends the entire forest to each worker on each iteration. This is because (a) the node queue is FIFO and (b) the closure references the entire array of trees (topNodes). (a) causes RFs to handle splits in many trees, especially early on in learning. (b) sends all trees explicitly.

Proposal:
(a) Change the RF node queue to be FILO, so that RFs tend to focus on 1 or a few trees before focusing on others.
(b) Change topNodes to pass only the trees required on that iteration.

Issue Links

• Is contained by: SPARK-14046 RandomForest improvement umbrella
• links to: [Github] Pull Request #14359 (jkbradley)