Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-6689 [Rust] [DataFusion] Query execution enhancements for 1.0.0 release
  3. ARROW-6659

[Rust] [DataFusion] Refactor of HashAggregateExec to support custom merge

    XMLWordPrintableJSON

    Details

      Description

      HashAggregateExec current creates one HashPartition per input partition for the initial aggregate per partition, and then explicitly calls MergeExec and then creates another HashPartition for the final reduce operation.

      This is fine for in-memory queries in DataFusion but is not extensible. For example, it is not possible to provide a different MergeExec implementation that would distribute queries to a cluster.

      A better design would be to move the logic into the query planner so that the physical plan contains explicit steps such as:

       

      - HashAggregate // final aggregate
        - MergeExec
          - HashAggregate // aggregate per partition
       

      This would then make it easier to customize the plan in other projects, to support distributed execution:

       - HashAggregate // final aggregate
         - MergeExec
            - DistributedExec
               - HashAggregate // aggregate per partition

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                kylemccarthy Kyle McCarthy
                Reporter:
                andygrove Andy Grove
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 40m
                  40m