Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-25650 Make analyzer rules used in once-policy idempotent
  3. SPARK-25690

Analyzer rule "HandleNullInputsForUDF" does not stabilize and can be applied infinitely

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.4.0
    • Fix Version/s: 2.4.0
    • Component/s: Spark Core, SQL
    • Labels:
      None

      Description

      This was fixed in SPARK-24891 and was then broken by SPARK-25044.

      The unit test in AnalysisSuite added in SPARK-24891 should have failed but didn't because it wasn't properly updated after the ScalaUDF constructor signature change. In the meantime, the other two end-to-end tests added in SPARK-24891 were shadowed by SPARK-24865.

      So the unit test mentioned above, if updated properly, can reproduce this issue:

      test("SPARK-24891 Fix HandleNullInputsForUDF rule") {
        val a = testRelation.output(0)
        val func = (x: Int, y: Int) => x + y
        val udf1 = ScalaUDF(func, IntegerType, a :: a :: Nil, nullableTypes = false :: false :: Nil)
        val udf2 = ScalaUDF(func, IntegerType, a :: udf1 :: Nil, nullableTypes = false :: false :: Nil)
        val plan = Project(Alias(udf2, "")() :: Nil, testRelation)
        comparePlans(plan.analyze, plan.analyze.analyze)
      }

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                maryannxue Wei Xue
                Reporter:
                maryannxue Wei Xue
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: