[SPARK-15574] Python meta-algorithms in Scala

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Component/s: ML, PySpark
• Assignee: Unassigned
• Reporter: Joseph K. Bradley
• Created: 26/May/16 18:12
• Updated: 26/Sep/17 08:40
• Resolved: 26/Sep/17 08:40

Description

This is an experimental idea for implementing Python ML meta-algorithms (CrossValidator, TrainValidationSplit, Pipeline, OneVsRest, etc.) in Scala. This would require a Scala wrapper for algorithms implemented in Python, somewhat analogous to Python UDFs.

The benefit of this change would be that we could avoid currently awkward conversions between Scala/Python meta-algorithms required for persistence. It would let us have full support for Python persistence and would generally simplify the implementation within MLlib.

Issue Links

• is related to: SPARK-21221 CrossValidator and TrainValidationSplit Persist Nested Estimators such as OneVsRest
• relates to: SPARK-17025 Cannot persist PySpark ML Pipeline model that includes custom Transformer
• relates to: SPARK-20099 Add transformSchema to pyspark.ml
• links to: [Github] Pull Request #13794 (yinxusen)