Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-12913

Reimplement stat functions as declarative function

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.0
    • Component/s: SQL
    • Labels:
      None

      Description

      As benchmarked and discussed here: https://github.com/apache/spark/pull/10786/files#r50038294.

      Benefits from codegen, the declarative aggregate function could be much faster than imperative one, we should re-implement all the builtin aggregate functions as declarative one.

      For skewness and kurtosis, we need to benchmark it to make sure that the declarative one is actually faster than imperative one.

        Issue Links

          Activity

          Hide
          apachespark Apache Spark added a comment -

          User 'davies' has created a pull request for this issue:
          https://github.com/apache/spark/pull/10960

          Show
          apachespark Apache Spark added a comment - User 'davies' has created a pull request for this issue: https://github.com/apache/spark/pull/10960
          Hide
          davies Davies Liu added a comment -

          Issue resolved by pull request 10960
          https://github.com/apache/spark/pull/10960

          Show
          davies Davies Liu added a comment - Issue resolved by pull request 10960 https://github.com/apache/spark/pull/10960

            People

            • Assignee:
              davies Davies Liu
              Reporter:
              davies Davies Liu
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development