[SPARK-48718] Got incastable error when deserializer in cogroup is resolved during application of DeduplicateRelation rule

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: SQL
• Assignee: Xinyi Yu
• Reporter: Xinyi Yu
• Created: 26/Jun/24 00:52
• Updated: 26/Jun/24 01:48
• Resolved: 26/Jun/24 01:48

Description

When running the following commands:

 

 

    val lhs = spark.createDataFrame(
      List(Row(123L)).asJava,
      StructType(Seq(StructField("GROUPING_KEY", LongType)))
    )
    val rhs = spark.createDataFrame(
      List(Row(0L, 123L)).asJava,
      StructType(Seq(StructField("ID", LongType), StructField("GROUPING_KEY", LongType)))
    )    val lhsKV = lhs.groupByKey((r: Row) => r.getAs[Long]("GROUPING_KEY"))
    val rhsKV = rhs.groupByKey((r: Row) => r.getAs[Long]("GROUPING_KEY"))
    val cogrouped = lhsKV.cogroup(rhsKV)(
      (a: Long, b: Iterator[Row], c: Iterator[Row]) => Iterator(0L)
    )
    val joined = rhs.join(cogrouped, col("ID") === col("value"), "left") 

 

 

It gets an error:

java.lang.ClassCastException: org.apache.spark.sql.catalyst.expressions.objects.AssertNotNull cannot be cast to org.apache.spark.sql.catalyst.analysis.UnresolvedDeserializer