Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-14220 Build and test Spark against Scala 2.12
  3. SPARK-25044

Address translation of LMF closure primitive args to Object in Scala 2.12

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotVotersStop watchingWatchersConvert to IssueLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.4.0
    • Fix Version/s: 2.4.0
    • Component/s: Spark Core, SQL
    • Labels:
      None

      Description

      A few SQL-related tests fail in Scala 2.12, such as UDFSuite's "SPARK-24891 Fix HandleNullInputsForUDF rule":

      - SPARK-24891 Fix HandleNullInputsForUDF rule *** FAILED ***
      Results do not match for query:
      ...
      == Results ==
      
      == Results ==
      !== Correct Answer - 3 == == Spark Answer - 3 ==
      !struct<> struct<a:bigint,b:int,c:int>
      ![0,10,null] [0,10,0]
      ![1,12,null] [1,12,1]
      ![2,14,null] [2,14,2] (QueryTest.scala:163)

      You can kind of get what's going on reading the test:

      test("SPARK-24891 Fix HandleNullInputsForUDF rule") {
      // assume(!ClosureCleanerSuite2.supportsLMFs)
      // This test won't test what it intends to in 2.12, as lambda metafactory closures
      // have arg types that are not primitive, but Object
      val udf1 = udf({(x: Int, y: Int) => x + y})
      val df = spark.range(0, 3).toDF("a")
      .withColumn("b", udf1($"a", udf1($"a", lit(10))))
      .withColumn("c", udf1($"a", lit(null)))
      val plan = spark.sessionState.executePlan(df.logicalPlan).analyzed
      
      comparePlans(df.logicalPlan, plan)
      checkAnswer(
      df,
      Seq(
      Row(0, 10, null),
      Row(1, 12, null),
      Row(2, 14, null)))
      }

       

      It seems that the closure that is fed in as a UDF changes behavior, in a way that primitive-type arguments are handled differently. For example an Int argument, when fed 'null', acts like 0.

      I'm sure it's a difference in the LMF closure and how its types are understood, but not exactly sure of the cause yet.

        Attachments

          Activity

          $i18n.getText('security.level.explanation', $currentSelection) Viewable by All Users
          Cancel

            People

            • Assignee:
              srowen Sean R. Owen Assign to me
              Reporter:
              srowen Sean R. Owen

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment