Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-32521

WithFields Expression should not be foldable

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.1.0
    • 3.1.0
    • SQL
    • None

    Description

      The following query currently fails on master brach: 

      sql("SELECT named_struct('a', 1, 'b', 2) a")
      .select($"a".withField("c", lit(3)).as("a"))
      .show(false)
      
      // java.lang.UnsupportedOperationException: Cannot evaluate expression: with_fields(named_struct(a, 1, b, 2), c, 3)
      

      This happens because the Catalyst optimizer tries to statically evaluate the WithFields Expression (via the ConstantFolding rule), however it cannot do so because WithFields Expression is Unevaluable

      The likely solution here is to change WithFields Expression so that it is not foldable under any circumstance. This should solve the issue because then Catalyst will no longer attempt to statically evaluate WithFields Expression anymore. 

       

       

      Attachments

        Activity

          People

            fqaiser94 fqaiser94
            fqaiser94 fqaiser94
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: