[SPARK-22198] Java incompatibility when extending UnaryTransformer or Transformer

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 2.2.0
• Component/s: Java API, ML
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Akos Tomasits
• Created: 04/Oct/17 07:41
• Updated: 21/May/19 04:14
• Resolved: 21/May/19 04:14

Description

It is not possible to create proper Java custom Transformer by extending UnaryTransformer or Transformer.

The built-in Params (e.g. defined in HasInputColumn trait) cannot be used and custom Params cannot be added.

It seems that the method 'uid()' is called during object creation before the provided 'uid' constructor parameter could be set.

This leads to the following error:

java.lang.IllegalArgumentException: requirement failed: Param <prefix>1563950936fa_inputCol does not belong to <prefix>_d4105b75c4aa.

If you extend UnaryTransformer and try to use it e.g. through CrossValidator, you will need to explicitly include a constructor, which receives a String parameter. As I saw in the source of built in transformers, this parameter is a 'uid', which should be set in the object. However, it is not possible to do it in time, because the uid() method is invoked (and its result might be used) before this constructor finishes.

Sample class:

public class TextCleaner extends UnaryTransformer<String, String, TextCleaner>
implements Serializable, DefaultParamsWritable, DefaultParamsReadable<TextCleaner> {

private static final long serialVersionUID = 2658543236303100458L;

private static final String sparkUidPrefix = "TextCleaner";

private final String sparkUid;

public TextCleaner()

Unknown macro: { sparkUid = org.apache.spark.ml.util.Identifiable$.MODULE$.randomUID(sparkUidPrefix); }

public TextCleaner(String uid)

Unknown macro: { sparkUid = uid; }

@Override
public String uid()

Unknown macro: { // This method is called by parent class, before object creation finishes return sparkUid; }

...

Issue Links

• relates to: SPARK-12606 Scala/Java compatibility issue Re: how to extend java transformer from Scala UnaryTransformer ?
• relates to: SPARK-15605 ML JavaDeveloperApiExample is broken