[SPARK-32459] UDF regression of WrappedArray supporting caused by SPARK-31826

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.0
• Fix Version/s: 3.1.0
• Component/s: SQL
• Assignee: wuyi
• Reporter: wuyi
• Created: 27/Jul/20 12:55
• Updated: 28/Jul/20 12:26
• Resolved: 28/Jul/20 12:26

Description

 

test("WrappedArray") {
  val myUdf = udf((a: WrappedArray[Int]) =>
    WrappedArray.make[Int](Array(a.head + 99)))
  checkAnswer(Seq(Array(1))
    .toDF("col")
    .select(myUdf(Column("col"))),
    Row(ArrayBuffer(100)))
}

Execute the above test in master branch, we'll hit the error:

[info]   org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0, 192.168.101.3, executor driver): java.lang.RuntimeException: Error while decoding: java.lang.ClassCastException: scala.collection.mutable.ArrayBuffer cannot be cast to scala.collection.mutable.WrappedArray[info]  

However, the test can be executed successfully in branch-3.0.

 

This's actually a regression caused by SPARK-31826. And the regression happens after we changed the catalyst-to-scala converter from CatalystTypeConverters to ExpressionEncoder.deserializer .

 

Issue Links

• is caused by: SPARK-31826 Support composed type of case class for typed Scala UDF
• links to: [Github] Pull Request #29261 (Ngone51)
• links to: [Github] Pull Request #29261 (Ngone51)