[SPARK-19422] Cache input data in algorithms

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 2.2.0
• Component/s: ML
• Assignee: Unassigned
• Reporter: Ruifeng Zheng
• Created: 01/Feb/17 08:36
• Updated: 14/Jun/18 09:14
• Resolved: 14/Jun/18 09:14

Description

Now some algorithms cache the input dataset if it was not cached any more StorageLevel.NONE:
FeedForwardTrainer, LogisticRegression, OneVsRest, KMeans, AFTSurvivalRegression, IsotonicRegression, LinearRegression with non-WSL solver

It maybe reasonable to cache input for others:
DecisionTreeClassifier, GBTClassifier, RandomForestClassifier, LinearSVC
BisectingKMeans, GaussianMixture, LDA
DecisionTreeRegressor, GBTRegressor, GeneralizedLinearRegression with IRLS solver, RandomForestRegressor

NaiveBayes is not included since it only make one pass on the data.
MultilayerPerceptronClassifier is not included since the data is cached in FeedForwardTrainer.train

Issue Links

• is blocked by: SPARK-18608 Spark ML algorithms that check RDD cache level for internal caching double-cache data
• is related to: SPARK-21972 Allow users to control input data persistence in ML Estimators via a handlePersistence ml.Param
• links to: [Github] Pull Request #16763 (zhengruifeng)