[BEAM-8316] What is corresponding data type to set in UDF parameter to match RecordType

• Type: Bug
• Status: Triage Needed
• Priority: P3
• Resolution: Unresolved
• Affects Version/s: 2.15.0
• Component/s: dsl-sql
• Assignee: Unassigned
• Reporter: Yang Zhang
• Created: 26/Sep/19 17:55
• Updated: 12/Jul/23 07:13

Description

Hello Beam community, 

I want to have an UDF to take a record as input. Per error info as shown below, it indicates that the input is RecordType, but ** what should I set in the UDF parameter so that Beam would not complain about the type compatibility? Below is the rull error trace. Thank you very much!

 

========error trace===========

Exception in thread "main" org.apache.beam.sdk.extensions.sql.impl.ParseException: Unable to parse query select fooudf(pv.header) from kafka.tracking.PageViewEvent as pvException in thread "main" org.apache.beam.sdk.extensions.sql.impl.ParseException: Unable to parse query select fooudf(pv.header) from kafka.tracking.PageViewEvent as pv at org.apache.beam.sdk.extensions.sql.impl.CalciteQueryPlanner.convertToBeamRel(CalciteQueryPlanner.java:165) at org.apache.beam.sdk.extensions.sql.impl.BeamSqlEnv.parseQuery(BeamSqlEnv.java:103) at org.apache.beam.sdk.extensions.sql.SqlTransform.expand(SqlTransform.java:124) at org.apache.beam.sdk.extensions.sql.SqlTransform.expand(SqlTransform.java:82) at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:539) at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:473) at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:44) at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:169) at com.linkedin.samza.sql.engine.BeamSqlEntry.preparePipeline(BeamSqlEntry.java:52) at com.linkedin.samza.sql.engine.BeamSqlEntry.exec(BeamSqlEntry.java:41) at com.linkedin.samza.sql.engine.BeamSqlUI.main(BeamSqlUI.java:33)Caused by: org.apache.calcite.tools.ValidationException: org.apache.calcite.runtime.CalciteContextException: From line 1, column 8 to line 1, column 24: No match found for function signature fooudf(<RecordType(INTEGER memberId, VARCHAR viewerUrn, VARCHAR applicationViewerUrn, VARCHAR csUserUrn, BIGINT time, VARCHAR server, VARCHAR service, VARCHAR environment, VARBINARY guid, VARBINARY treeId, INTEGER requestId, VARCHAR impersonatorId, VARCHAR version, VARCHAR instance, VARCHAR appName, VARCHAR testId, VARCHAR testSegmentId, RecordType(BIGINT time, VARCHAR server, VARCHAR instance, VARCHAR appName, VARBINARY messageId, INTEGER auditVersion, VARCHAR fabricUrn, VARCHAR clusterConnectionString) auditHeader, RecordType(VARCHAR pageUrn, VARBINARY trackingId) pageInstance, RecordType(VARCHAR applicationUrn, VARCHAR version, VARBINARY trackingId) clientApplicationInstance, VARCHAR originSource, VARCHAR sessionUrn, RecordType(VARBINARY treeId, INTEGER requestId, INTEGER taskId, VARCHAR rpcTrace, BOOLEAN forceTraceEnabled, (VARCHAR, VARCHAR) MAP context, DOUBLE scaleFactor) traceData)>) at org.apache.calcite.prepare.PlannerImpl.validate(PlannerImpl.java:190) at org.apache.beam.sdk.extensions.sql.impl.CalciteQueryPlanner.convertToBeamRel(CalciteQueryPlanner.java:136) ... 10 moreCaused by: org.apache.calcite.runtime.CalciteContextException: From line 1, column 8 to line 1, column 24: No match found for function signature fooudf(<RecordType(INTEGER memberId, VARCHAR viewerUrn, VARCHAR applicationViewerUrn, VARCHAR csUserUrn, BIGINT time, VARCHAR server, VARCHAR service, VARCHAR environment, VARBINARY guid, VARBINARY treeId, INTEGER requestId, VARCHAR impersonatorId, VARCHAR version, VARCHAR instance, VARCHAR appName, VARCHAR testId, VARCHAR testSegmentId, RecordType(BIGINT time, VARCHAR server, VARCHAR instance, VARCHAR appName, VARBINARY messageId, INTEGER auditVersion, VARCHAR fabricUrn, VARCHAR clusterConnectionString) auditHeader, RecordType(VARCHAR pageUrn, VARBINARY trackingId) pageInstance, RecordType(VARCHAR applicationUrn, VARCHAR version, VARBINARY trackingId) clientApplicationInstance, VARCHAR originSource, VARCHAR sessionUrn, RecordType(VARBINARY treeId, INTEGER requestId, INTEGER taskId, VARCHAR rpcTrace, BOOLEAN forceTraceEnabled, (VARCHAR, VARCHAR) MAP context, DOUBLE scaleFactor) traceData)>) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.calcite.runtime.Resources$ExInstWithCause.ex(Resources.java:463) at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:787) at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:772) at org.apache.calcite.sql.validate.SqlValidatorImpl.newValidationError(SqlValidatorImpl.java:4825) at org.apache.calcite.sql.validate.SqlValidatorImpl.handleUnresolvedFunction(SqlValidatorImpl.java:1739) at org.apache.calcite.sql.SqlFunction.deriveType(SqlFunction.java:270) at org.apache.calcite.sql.SqlFunction.deriveType(SqlFunction.java:215) at org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:5584) at org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:5571) at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:138) at org.apache.calcite.sql.validate.SqlValidatorImpl.deriveTypeImpl(SqlValidatorImpl.java:1657) at org.apache.calcite.sql.validate.SqlValidatorImpl.deriveType(SqlValidatorImpl.java:1642) at org.apache.calcite.sql.validate.SqlValidatorImpl.expandSelectItem(SqlValidatorImpl.java:462) at org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelectList(SqlValidatorImpl.java:4089) at org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect(SqlValidatorImpl.java:3352) at org.apache.calcite.sql.validate.SelectNamespace.validateImpl(SelectNamespace.java:60) at org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84) at org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:994) at org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:954) at org.apache.calcite.sql.SqlSelect.validate(SqlSelect.java:216) at org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:929) at org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:633) at org.apache.calcite.prepare.PlannerImpl.validate(PlannerImpl.java:188) ... 11 moreCaused by: org.apache.calcite.sql.validate.SqlValidatorException: No match found for function signature fooudf(<RecordType(INTEGER memberId, VARCHAR viewerUrn, VARCHAR applicationViewerUrn, VARCHAR csUserUrn, BIGINT time, VARCHAR server, VARCHAR service, VARCHAR environment, VARBINARY guid, VARBINARY treeId, INTEGER requestId, VARCHAR impersonatorId, VARCHAR version, VARCHAR instance, VARCHAR appName, VARCHAR testId, VARCHAR testSegmentId, RecordType(BIGINT time, VARCHAR server, VARCHAR instance, VARCHAR appName, VARBINARY messageId, INTEGER auditVersion, VARCHAR fabricUrn, VARCHAR clusterConnectionString) auditHeader, RecordType(VARCHAR pageUrn, VARBINARY trackingId) pageInstance, RecordType(VARCHAR applicationUrn, VARCHAR version, VARBINARY trackingId) clientApplicationInstance, VARCHAR originSource, VARCHAR sessionUrn, RecordType(VARBINARY treeId, INTEGER requestId, INTEGER taskId, VARCHAR rpcTrace, BOOLEAN forceTraceEnabled, (VARCHAR, VARCHAR) MAP context, DOUBLE scaleFactor) traceData)>) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.calcite.runtime.Resources$ExInstWithCause.ex(Resources.java:463) at org.apache.calcite.runtime.Resources$ExInst.ex(Resources.java:572) ... 33 more