Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-41141

avoid introducing a new aggregate expression in the analysis phase when subquery is referencing it

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: In Progress
    • Minor
    • Resolution: Unresolved
    • 3.3.1
    • None
    • SQL

    Description

      Currently the  analyzer phase rules on subquery referencing the aggregate expression in outer query, avoids introducing a new aggregate only for a single level aggregate function. It introduces new aggregate expression for nested aggregate functions.

      It is possible to avoid  adding this extra aggregate expression  easily, atleast if the outer projection involving aggregate function is exactly same as the one that is used in subquery, or if the outer query's projection involving aggregate function is a subtree of the subquery's expression.

       

      Thus consider the following 2 cases:

      1) select  cos (sum(a)) , b from t1  group by b having exists (select x from t2 where y = cos(sum(a)) )

      2) select  sum(a) , b from t1  group by b having exists (select x from t2 where y = cos(sum(a)) )

       

      In both the above cases, there is no need for adding extra aggregate expression.

       

      I am also investigating if its possible to avoid if the case is 

       

      3) select  Cos(sum(a)) , b from t1  group by b having exists (select x from t2 where y = sum(a) )

       

      This Jira also is needed for another issue where subquery datasource v2  is projecting columns which are not needed. ( no Jira filed yet for that, will do that..)

       

      Will be opening a PR for this soon..

      Attachments

        Activity

          People

            Unassigned Unassigned
            ashahid7 Asif
            Peter Toth Peter Toth
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated: