[ARROW-16410] [C++] Scanner -> ScanNode

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: C++
• External issue URL: https://github.com/apache/arrow/issues/31787
• Assignee: Unassigned
• Reporter: Weston Pace
• Created: 28/Apr/22 19:31
• Updated: 11/Jan/23 11:43

Description

This is going to be a parent issue for a number of changes I'd like to make to the scanner.

• I'd like to remove the concept of the scanner from the public API
• Related to that, ScanOptions will probably break into two classes. QueryOptions will be the public facing half while ScanOptions will be options sent to the ScanNode. Most users won't see ScanOptions as it will be internal.
• For example, QueryOptions will have batch_readahead which represents how many "query engine batches" to readahead. Since files & the query engine have different ideas of what constitutes a "batch" the related property in ScanOptions will be rows_to_readahead.
• Another example is projection, in QueryOptions projection is column selection as well as custom projection expressions that a user wants to run. In ScanOptions "projection" is the desired list of columns and the output type for each column, which controls casting and inference.
• Partially related (and partially unrelated) to the above two items I would like to move the scanner away from AsyncGenerator and recast it as an execution engine node.
• The Scanner class will become deprecated and eventually go away. Some methods like Scanner::ToTable may move into a new QueryBuilder or Query object.

Issue Links

• depends upon: ARROW-12311 [Python][R] Expose (hide?) ScanOptions
• depends upon: ARROW-12559 [C++] Add support for Future<MoveOnlyDataType>
• depends upon: ARROW-16072 [C++] Migrate scanner logic to ExecPlan, remove merged generator
• depends upon: ARROW-16409 [C++][Python][R] Deprecate "scanner" (but keep "scan node") from public API