[SPARK-11618] Refactoring of basic ML import/export

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.6.0
• Component/s: ML
• Target Version/s: 1.6.0
• Assignee: Joseph K. Bradley
• Reporter: Joseph K. Bradley
• Shepherd: Xiangrui Meng
• Created: 10/Nov/15 05:36
• Updated: 10/Feb/16 12:26
• Resolved: 10/Nov/15 19:36

Description

This is for a few updates to the original PR for basic ML import/export in SPARK-11217.

• The original PR diverges from the design doc in that it does not include the Spark version or a model format version. We should include the Spark version in the metadata. If we do that, then we don't really need a model format version.
• Proposal: DefaultParamsWriter includes two separable pieces of logic in save(): (a) handling overwriting and (b) saving Params. I want to separate these by putting (a) in a save() method in Writer which calls an abstract saveImpl, and (b) in the saveImpl implementation in DefaultParamsWriter. This is described below:

abstract class Writer {
  def save(path: String) = {
    // handle overwrite
    saveImpl(path)
  }
  def saveImpl(path: String)   // abstract
}

class DefaultParamsWriter extends Writer {
  def saveImpl(path: String) = {
    // save Params
  }
}

Issue Links

• is related to: SPARK-13265 Refactoring of basic ML import/export for other file system besides HDFS
• links to: [Github] Pull Request #9587 (jkbradley)