[SPARK-25690] Analyzer rule "HandleNullInputsForUDF" does not stabilize and can be applied infinitely

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 2.4.0
• Component/s: Spark Core, SQL
• Assignee: Wei Xue
• Reporter: Wei Xue
• Created: 09/Oct/18 19:32
• Updated: 12/Oct/18 03:45
• Resolved: 12/Oct/18 03:45

Description

This was fixed in SPARK-24891 and was then broken by SPARK-25044.

The unit test in AnalysisSuite added in SPARK-24891 should have failed but didn't because it wasn't properly updated after the ScalaUDF constructor signature change. In the meantime, the other two end-to-end tests added in SPARK-24891 were shadowed by SPARK-24865.

So the unit test mentioned above, if updated properly, can reproduce this issue:

test("SPARK-24891 Fix HandleNullInputsForUDF rule") {
  val a = testRelation.output(0)
  val func = (x: Int, y: Int) => x + y
  val udf1 = ScalaUDF(func, IntegerType, a :: a :: Nil, nullableTypes = false :: false :: Nil)
  val udf2 = ScalaUDF(func, IntegerType, a :: udf1 :: Nil, nullableTypes = false :: false :: Nil)
  val plan = Project(Alias(udf2, "")() :: Nil, testRelation)
  comparePlans(plan.analyze, plan.analyze.analyze)
}

Issue Links

• is a clone of: SPARK-25044 Address translation of LMF closure primitive args to Object in Scala 2.12
• links to: [Github] Pull Request #22701 (maryannxue)