Description
This is an experimental idea for implementing Python ML meta-algorithms (CrossValidator, TrainValidationSplit, Pipeline, OneVsRest, etc.) in Scala. This would require a Scala wrapper for algorithms implemented in Python, somewhat analogous to Python UDFs.
The benefit of this change would be that we could avoid currently awkward conversions between Scala/Python meta-algorithms required for persistence. It would let us have full support for Python persistence and would generally simplify the implementation within MLlib.
Attachments
Issue Links
- is related to
-
SPARK-21221 CrossValidator and TrainValidationSplit Persist Nested Estimators such as OneVsRest
-
- Resolved
-
- relates to
-
SPARK-17025 Cannot persist PySpark ML Pipeline model that includes custom Transformer
-
- Resolved
-
-
SPARK-20099 Add transformSchema to pyspark.ml
-
- Resolved
-
- links to