[ARROW-11606] [Rust] [DataFusion] Need guidance on HashAggregateExec reconstruction

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.0.0
• Component/s: Rust - DataFusion
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/27473
• Assignee: Andy Grove
• Reporter: Andy Grove
• Created: 12/Feb/21 01:37
• Updated: 11/Jan/23 08:20
• Resolved: 12/Feb/21 22:28

Description

We have run into an issue in the Ballista project where we are reconstructing the Final and Partial HashAggregateExec operators [1] for distributed execution and we need some guidance.

The Partial HashAggregateExec gets created OK and executes correctly.

However, when we create the Final HashAggregateExec, it is not finding the expected schema in the input operator. The partial exec outputs field names ending with "[sum]" and "[count]" and so on but the final aggregate doesn't seem to be looking for those names.

It is also worth noting that the Final and Partial executors are not connected directly in this usage.

The Partial exec is executed and output streamed to disk.

The Final exec then runs against the output from the Partial exec.

We may need to make changes in DataFusion to allow other crates to support this kind of use case?

 [1] https://github.com/ballista-compute/ballista/pull/491

 

Issue Links

• links to: GitHub Pull Request #9481