[ARROW-12231] [C++][Dataset] Separate datasets backed by readers from InMemoryDataset

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 5.0.0
• Component/s: C++
• Labels: dataset datasets pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/28047
• Assignee: David Li
• Reporter: Weston Pace
• Created: 06/Apr/21 15:31
• Updated: 11/Jan/23 08:25
• Resolved: 06/May/21 17:10

Description

From ARROW-10882/https://github.com/apache/arrow/pull/9802 

• Backing an InMemoryDataset with a reader is misleading. Let's split that out into a separate class.
• Dataset scanning can then use an I/O thread for the new class. (Note that for Python, we'll need to be careful to release the GIL before any operations so that the I/O thread can acquire the GIL to call into the underlying Python reader/file object.)
• Longer-term, we should interface with Python's async.

Issue Links

• relates to: ARROW-10882 [Python][Dataset] Writing dataset from python iterator of record batches
• links to: GitHub Pull Request #10070