[SPARK-3727] Trees and ensembles: More prediction functionality

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Component/s: MLlib
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Joseph K. Bradley
• Created: 29/Sep/14 18:40
• Updated: 08/Oct/19 05:42
• Resolved: 08/Oct/19 05:42

Description

DecisionTree and RandomForest currently predict the most likely label for classification and the mean for regression. Other info about predictions would be useful.

For classification: estimated probability of each possible label
For regression: variance of estimate

RandomForest could also create aggregate predictions in multiple ways:

• Predict mean or median value for regression.
• Compute variance of estimates (across all trees) for both classification and regression.

Issue Links

• is blocked by: SPARK-6113 Stabilize DecisionTree and ensembles APIs
• is blocked by: SPARK-7131 Move tree,forest implementation from spark.mllib to spark.ml
• is duplicated by: SPARK-4736 functions returning the category with weights
• is related to: SPARK-4240 Refine Tree Predictions in Gradient Boosting to Improve Prediction Accuracy.

Sub-Tasks

• Decision trees: predict class probabilities: Resolved
• Random forest: predict class probabilities: Closed
• Make random forest classifier extend Classifier abstraction: Resolved
• DecisionTreeRegressor: provide variance of prediction: Resolved
• RandomForestRegressor: provide variance of predictions: Resolved