[BEAM-3739] @Parameter annotation does not work for UDFs in Beam SQL

• Type: Bug
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Affects Version/s: 2.3.0
• Fix Version/s: 2.4.0
• Component/s: dsl-sql
• Assignee: Mingmin Xu
• Reporter: Samuel Waggoner
• Created: 23/Feb/18 16:21
• Updated: 24/Jul/20 20:11
• Resolved: 27/Feb/18 17:15

Description

BeamSqlUdf javadoc indicates you can have optional parameters, but this functionality is not working. I implemented the following copy/paste from the doc https://beam.apache.org/documentation/sdks/javadoc/2.3.0/org/apache/beam/sdk/extensions/sql/BeamSqlUdf.html:

public static class MyLeftFunction implements BeamSqlUdf {
 public String eval(
 @Parameter(name = "s") String s,
 @Parameter(name = "n", optional = true) Integer n) {
 return s.substring(0, n == null ? 1 : n);
 }
}

I modify a query in BeamSqlExample.java to use it. With all parameters supplied, it completes successfully:

//Case 1. run a simple SQL query over input PCollection with BeamSql.simpleQuery;
PCollection<Row> outputStream = inputTable.apply(
    BeamSql.query("select c1, leftfn('string1', 1) as c2, c3 from PCOLLECTION where c1 > 1")
        .registerUdf("leftfn", MyLeftFunction.class));

With the optional parameter left off, I get an exception:

//Case 1. run a simple SQL query over input PCollection with BeamSql.simpleQuery;
PCollection<Row> outputStream = inputTable.apply(
 BeamSql.query("select c1, leftfn('string1') as c2, c3 from PCOLLECTION where c1 > 1")
 .registerUdf("leftfn", MyLeftFunction.class));

Exception in thread "main" java.lang.IllegalStateException: java.lang.UnsupportedOperationException: Operator: DEFAULT is not supported yet!
 at org.apache.beam.sdk.extensions.sql.QueryTransform.expand(QueryTransform.java:75)
 at org.apache.beam.sdk.extensions.sql.QueryTransform.expand(QueryTransform.java:47)
 at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537)
 at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:472)
 at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:286)
 at org.apache.beam.sdk.extensions.sql.example.BeamSqlExample.main(BeamSqlExample.java:76)
Caused by: java.lang.UnsupportedOperationException: Operator: DEFAULT is not supported yet!
 at org.apache.beam.sdk.extensions.sql.impl.interpreter.BeamSqlFnExecutor.buildExpression(BeamSqlFnExecutor.java:424)
 at org.apache.beam.sdk.extensions.sql.impl.interpreter.BeamSqlFnExecutor.buildExpression(BeamSqlFnExecutor.java:201)
 at org.apache.beam.sdk.extensions.sql.impl.interpreter.BeamSqlFnExecutor.<init>(BeamSqlFnExecutor.java:125)
 at org.apache.beam.sdk.extensions.sql.impl.rel.BeamProjectRel.buildBeamPipeline(BeamProjectRel.java:70)
 at org.apache.beam.sdk.extensions.sql.QueryTransform.expand(QueryTransform.java:73)
 ... 5 more

Issue Links

• links to: GitHub Pull Request #4753