Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-18186

Migrate HiveUDAFFunction to TypedImperativeAggregate for partial aggregation support

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.6.2, 2.0.1
    • 2.2.0
    • SQL
    • None

    Description

      Currently, Hive UDAFs in Spark SQL don't support partial aggregation. Any query involving any Hive UDAFs has to fall back to SortAggregateExec without partial aggregation.

      This issue can be fixed by migrating HiveUDAFFunction to TypedImperativeAggregate, which already provides partial aggregation support for aggregate functions that may use arbitrary Java objects as aggregation states.

      Attachments

        Activity

          People

            lian cheng Cheng Lian
            lian cheng Cheng Lian
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: