Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-41805

Reuse expressions in WindowSpecDefinition

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.4.0
    • 3.4.0
    • SQL
    • None

    Description

      For complex expressions in window spec definition, we extract it and replace it with an AttributeReference (with an internal column name, e.g. "_w0").
      We can reuse the extracted expressions to avoid generator unnecessary Window node.

      For example: the logical plan of query

      SELECT max(value) over(partition by (key) order by cast(ts as timestamp)) as w1,
             min(value) over(partition by (key) order by cast(ts as timestamp)) as w2
      FROM t1
      

      contains two Window node.

      Attachments

        Activity

          People

            wankun Wan Kun
            wankun Wan Kun
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: