[ARROW-11068] [Rust] [DataFusion] Wrap more operators in CoalesceBatchExec

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Invalid
• Component/s: Rust - DataFusion
• External issue URL: https://github.com/apache/arrow/issues/26982
• Assignee: Andy Grove
• Reporter: Andy Grove
• Created: 29/Dec/20 18:39
• Updated: 11/Jan/23 08:17
• Resolved: 26/Apr/21 13:25

Description

Once https://github.com/apache/arrow/pull/9043 is merged, we should extend this to wrap HashJoinExec and HashAggregateExec as well since they can both produce small batches.

Rather than hard-code a list of operators that need to be wrapped, we should find a more generic mechanism so that plans can declare if their input and/or output batches should be coalesced (similar to how we handle partitioning) and this would allow custom operators outside of DataFusion to benefit from this optimization.