[SPARK-38485] Non-deterministic UDF executed multiple times when combined with withField

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.3.0
• Component/s: SQL
• Labels: Correctness
• Assignee: Unassigned
• Reporter: Tanel Kiis
• Created: 09/Mar/22 18:11
• Updated: 07/Jun/22 01:33

Description

When adding fields to a result of a non-deterministic UDF, that returns a struct, then that UDF is executed multiple times (once per field) for each row.

In this UT df1 passes, but df2 fails with something like:
"279751724 did not equal -1023188908"

  test("SPARK-XXXXX: non-deterministic UDF should be called once when adding fields") {
    val nondeterministicUDF = udf((s: Int) => {
      val r = Random.nextInt()
      // Both values should be the same
      GroupByKey(r, r)
    }).asNondeterministic()

    val df1 = spark.range(5).select(nondeterministicUDF($"id"))
    df1.collect().foreach {
      row => assert(row.getStruct(0).getInt(0) == row.getStruct(0).getInt(1))
    }

    val df2 = spark.range(5).select(nondeterministicUDF($"id").withField("new", lit(7)))
    df2.collect().foreach {
      row => assert(row.getStruct(0).getInt(0) == row.getStruct(0).getInt(1))
    }
  }