[SPARK-6113] Stabilize DecisionTree and ensembles APIs

• Type: Sub-task
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.4.0
• Fix Version/s: 1.4.0
• Component/s: MLlib, PySpark
• Target Version/s: 1.4.0
• Assignee: Joseph K. Bradley
• Reporter: Joseph K. Bradley
• Created: 02/Mar/15 18:21
• Updated: 07/Jul/15 04:26
• Resolved: 25/Apr/15 19:27

Description

Issue: The APIs for DecisionTree and ensembles (RandomForests and GradientBoostedTrees) have been experimental for a long time. The API has become very convoluted because trees and ensembles have many, many variants, some of which we have added incrementally without a long-term design.

Proposal: This JIRA is for discussing changes required to finalize the APIs. After we discuss, I will make a PR to update the APIs and make them non-Experimental. This will require making many breaking changes; see the design doc for details.

Design doc : This outlines current issues and the proposed API.

Issue Links

• blocks: SPARK-3727 Trees and ensembles: More prediction functionality
• blocks: SPARK-7126 For spark.ml Classifiers, automatically index labels if they are not yet indexed
• blocks: SPARK-7131 Move tree,forest implementation from spark.mllib to spark.ml
• blocks: SPARK-7127 Broadcast spark.ml tree ensemble models for predict
• blocks: SPARK-7132 Add fit with validation set to spark.ml GBT
• contains: SPARK-3164 Store DecisionTree Split.categories as Set
• is blocked by: SPARK-5886 Add StringIndexer
• is blocked by: SPARK-4081 Categorical feature indexing
• is duplicated by: SPARK-5399 tree Losses strings should match loss names
• relates to: SPARK-7047 Model parent should be Optional
• links to: [Github] Pull Request #5009 (jkbradley)
• links to: [Github] Pull Request #5530 (jkbradley)
• links to: [Github] Pull Request #5567 (jkbradley)
• links to: [Github] Pull Request #5626 (jkbradley)