[SPARK-4587] Model export/import

• Type: Umbrella
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Component/s: ML, MLlib
• Assignee: Joseph K. Bradley
• Reporter: Xiangrui Meng
• Created: 25/Nov/14 04:26
• Updated: 11/Dec/16 05:05
• Resolved: 11/Dec/16 05:05

Description

This is an umbrella JIRA for one of the most requested features on the user mailing list. Model export/import can be done via Java serialization. But it doesn't work for models stored distributively, e.g., ALS and LDA. Ideally, we should provide save/load methods to every model. PMML is an option but it has its limitations. There are couple things we need to discuss: 1) data format, 2) how to preserve partitioning, 3) data compatibility between versions and language APIs, etc.

UPDATE: Design doc for model import/export

This document sketches machine learning model import/export plans, including goals, an API, and development plans.

UPDATE: As in the design doc, we plan to support:

• Our own Spark-specific format.
  • This is needed to (a) support distributed models and (b) get model import/export support into Spark quickly (while avoiding the complexity of PMML).
• PMML
  • This is needed since it is the most commonly used format in industry.

This JIRA will be for the internal Spark-specific format described in the design doc. Parallel JIRAs will cover PMML.

Issue Links

• is depended upon by: SPARK-5991 Python API for ML model import/export
• is depended upon by: SPARK-18305 CLONE - Python API for ML model import/export
• is duplicated by: SPARK-5359 ML model import/export
• is related to: SPARK-1406 PMML model evaluation support via MLib
• is related to: SPARK-8545 PMML improvement umbrella
• relates to: SPARK-6725 Model export/import for Pipeline API (Scala)
• links to: [Github] Pull Request #4233 (jkbradley)
• links to: [Github] Pull Request #4816 (jkbradley)

Sub-Tasks

• Model import/export for GLMs and Naive Bayes: Resolved
• Model import/export for DecisionTree and ensembles: Resolved
• Model import/export for ALS: Resolved
• Model import/export for Word2Vec: Resolved
• Model import/export for KMeansModel: Resolved
• Model import/export for GaussianMixtureModel: Resolved
• Model import/export for PowerIterationClusteringModel: Resolved
• Model import/export for LDAModel: Resolved
• Model import/export for IsotonicRegression: Resolved
• Model import/export for StreamingKMeansModel: Closed
• Model import/export for ChiSqSelector: Resolved
• Model import/export for FPGrowth: Resolved
• Model import/export for PrefixSpan: Resolved