[ARROW-11331] [Rust][DataFusion] Improve performance of Array.slice

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Rust, Rust - DataFusion
• External issue URL: https://github.com/apache/arrow/issues/27229
• Assignee: Unassigned
• Reporter: Daniël Heres
• Created: 20/Jan/21 20:12
• Updated: 11/Jan/23 08:19

Description

In DataFusion we are using Array.slice since https://github.com/apache/arrow/pull/9271 to pass data into the accumulators, instead of having the overhead of building arrays (possibly with few rows) at once.

However, it seems pretty inefficient by now (taking a 1/6 of instructions for hash aggregates) doing some allocations under the hood instead of the promised "zero copy", much more than for example take which copies / shuffles the entire array based on indices.

jorgecarleitao

Yes, slicing is suboptimal atm. Also, IMO it should not be the Array to implement that method, but each implementation individually. I haven't touch that part yet, though.