Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-35448 Subexpression elimination enhancements
  3. SPARK-35449

Should not extract common expressions from value expressions when elseValue is empty in CaseWhen

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.1.1, 3.2.0
    • 3.2.0, 3.1.3
    • SQL
    • None

    Description

      https://github.com/apache/spark/pull/30245  added support for creating subexpressions that are present in all branches of conditional statements. However, for a statement to be in "all branches" of a CaseWhen statement, it must also be in the elseValue. This can lead to a subexpression to be created and run for branches of a conditional that don't pass. This can cause issues especially with a UDF in a branch that gets executed assuming the condition is true. For example:

      val col = when($"id" < 0, myUdf($"id"))
      spark.range(1).select(when(col > 0, col)).show()
      

      myUdf($"id") gets extracted as a subexpression and executed even though both conditions don't pass and it should never be executed.

      Attachments

        Activity

          People

            Kimahriman Adam Binford
            viirya L. C. Hsieh
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: