[SPARK-32154] Use ExpressionEncoder for the return type of ScalaUDF to convert to catalyst type

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.0
• Fix Version/s: 3.1.0
• Component/s: SQL
• Assignee: wuyi
• Reporter: wuyi
• Created: 02/Jul/20 08:51
• Updated: 28/Mar/21 17:44
• Resolved: 12/Jul/20 00:46

Description

Users now could register a UDF with Instant/LocalDate as return type even with 

spark.sql.datetime.java8API.enabled=false. However, the UDF can only be really used with spark.sql.datetime.java8API.enabled=true. This could make users confused.

The problem is we use ExpressionEncoder to ser/deser types when registering the UDF, but use Catalyst converters to ser/deser types, which is under control of  spark.sql.datetime.java8API.enabled,  when executing UDF.

If we could also use ExpressionEncoder to ser/deser types, similar to what we do for input parameter types, the, UDF could support Instant/LocalDate, event other combined complex types as well.

Issue Links

• causes: SPARK-34002 Broken UDF Encoding
• is related to: SPARK-34829 transform_values return identical values when it's used with udf that returns reference type
• links to: [Github] Pull Request #28979 (Ngone51)
• links to: [Github] Pull Request #28979 (Ngone51)