[SPARK-17025] Cannot persist PySpark ML Pipeline model that includes custom Transformer

• Type: New Feature
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 2.3.0
• Component/s: ML, PySpark
• Assignee: Ajay Saini
• Reporter: Nicholas Chammas
• Created: 11/Aug/16 19:20
• Updated: 04/Jun/19 23:12
• Resolved: 22/Jun/18 18:19

Description

Following the example in this Databricks blog post under "Python tuning", I'm trying to save an ML Pipeline model.

This pipeline, however, includes a custom transformer. When I try to save the model, the operation fails because the custom transformer doesn't have a _to_java attribute.

Traceback (most recent call last):
  File ".../file.py", line 56, in <module>
    model.bestModel.save('model')
  File "/usr/local/Cellar/apache-spark/2.0.0/libexec/python/lib/pyspark.zip/pyspark/ml/pipeline.py", line 222, in save
  File "/usr/local/Cellar/apache-spark/2.0.0/libexec/python/lib/pyspark.zip/pyspark/ml/pipeline.py", line 217, in write
  File "/usr/local/Cellar/apache-spark/2.0.0/libexec/python/lib/pyspark.zip/pyspark/ml/util.py", line 93, in __init__
  File "/usr/local/Cellar/apache-spark/2.0.0/libexec/python/lib/pyspark.zip/pyspark/ml/pipeline.py", line 254, in _to_java
AttributeError: 'PeoplePairFeaturizer' object has no attribute '_to_java'

Looking at the source code for ml/base.py, I see that not even the base Transformer class has such an attribute.

I'm assuming this is missing functionality that is intended to be patched up (i.e. like this).

I'm not sure if there is an existing JIRA for this (my searches didn't turn up clear results).

Issue Links

• is related to: SPARK-21542 Helper functions for custom Python Persistence
• is related to: SPARK-24632 Allow 3rd-party libraries to use pyspark.ml abstractions for Java wrappers for persistence
• is related to: SPARK-15574 Python meta-algorithms in Scala
• relates to: SPARK-11939 PySpark support model export/import for Pipeline API
• links to: PR 18888