[ARROW-7916] [C++][Dataset] Project IPC record batches to materialized fields

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.16.0
• Fix Version/s: 0.17.0
• Component/s: C++
• Labels: dataset pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/24137
• Assignee: Ben Kietzman
• Reporter: Ben Kietzman
• Created: 21/Feb/20 18:09
• Updated: 11/Jan/23 07:56
• Resolved: 27/Feb/20 19:21

Description

If batches mmaped from disk are projected before post filtering, unreferenced columns will never be accessed (so the memory map shouldn't do I/O on them).

At the same time, it'd probably be wise to explicitly document that batches yielded directly from fragments rather than from a Scanner will not be filtered or projected (so they will not match the fragment's schema and will include columns referenced by the filter even if they were not projected).

Issue Links

• links to: GitHub Pull Request #6474