[SPARK-5874] How to improve the current ML pipeline API?

• Type: Brainstorming
• Status: Resolved
• Priority: Critical
• Resolution: Done
• Component/s: ML
• Assignee: Xiangrui Meng
• Reporter: Xiangrui Meng
• Created: 17/Feb/15 21:18
• Updated: 05/May/18 03:11
• Resolved: 05/May/18 03:11

Description

I created this JIRA to collect feedbacks about the ML pipeline API we introduced in Spark 1.2. The target is to graduate this set of APIs in 1.4 with confidence, which requires valuable input from the community. I'll create sub-tasks for each major issue.

Design doc (WIP): https://docs.google.com/a/databricks.com/document/d/1plFBPJY_PriPTuMiFYLSm7fQgD1FieP4wt3oMVKMGcc/edit#

Issue Links

• incorporates: SPARK-8984 Developer documentation for ML Pipelines
• is related to: SPARK-14033 Merging Estimator & Model
• is related to: SPARK-10413 ML models should support prediction on single instances
• is related to: SPARK-7146 Should ML sharedParams be a public API?
• is related to: SPARK-11107 spark.ml should support more input column types: umbrella
• relates to: SPARK-6725 Model export/import for Pipeline API (Scala)
• relates to: SPARK-19498 Discussion: Making MLlib APIs extensible for 3rd party libraries

Sub-Tasks

• Transformer/Estimator should be copyable.: Resolved
• Better handling of default parameter values.: Resolved
• Move shared params to param.shared and use code gen: Resolved
• Better handling of pipeline parameters in PySpark: Resolved
• Enforce Params.validate by making it abstract: Resolved
• Add validation functionality to individual Param: Resolved
• Python: Add validation functionality to individual Param: Closed
• Python: Transformer/Estimator should be copyable: Resolved
• DSL for Pipeline assembly: Resolved
• Remove spark.ml Model, and have all Transformers have parent: Closed
• Add single- and multi-value support to ML Transformers: Resolved
• Should Evaluator be a PipelineStage: Resolved
• Add optional caching to Pipelines: Resolved
• Should ML Models contains single models or Pipelines?: Resolved
• Warm-start support for ML estimator: Resolved