[ARROW-11687] [Rust][DataFusion] RepartitionExec Hanging

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.1, 4.0.0
• Component/s: Rust - DataFusion
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/27550
• Assignee: Mike Seddon
• Reporter: Mike Seddon
• Created: 18/Feb/21 07:10
• Updated: 11/Jan/23 08:21
• Resolved: 21/Feb/21 10:39

Description

Found an interesting defect where the final partition of the `RepartitionExec::execute` thread spawner was consistently not being spawned via `tokio::spawn`. This meant that `RepartitionStream::poll_next` was sitting waiting forever for data that never arrived.

It looks like a race condition where the `JoinHandle` was not being `await`ed and something strange going on with the internals of tokio like lazy evaluation?

Issue Links

• links to: GitHub Pull Request #9523
• links to: GitHub Pull Request #9603