[ARROW-12311] [Python][R] Expose (hide?) ScanOptions

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Python, R
• External issue URL: https://github.com/apache/arrow/issues/28113
• Assignee: Unassigned
• Reporter: Weston Pace
• Created: 09/Apr/21 10:15
• Updated: 11/Jan/23 08:25

Description

Currently R completely hides the `ScanOptions` class.

In python the class is exposed but the documentation prefers `dataset.scan` (which hides both the scanner and the scan options).

However, there is some useful information in the `ScanOptions`.  Specifically, the projected schema (which is a product of the dataset schema and the projection expression and not easily recreated) and the materialized fields (the list of fields referenced by either the filter or the projection) which might be useful for reporting purposes.

Currently R uses the projected schema to convert a list of column names into a partition schema.  Python does not rely on either field.

 

Options:

 - Keep the status quo

 - Expose the ScanOptions object (which itself is exposed via the Scanner)

 - Expose the interesting fields via the Scanner

 

Currently the C++ design is halfway between the latter two (projected schema is exposed and options).  My preference would be the third option.  It raises a further question about how to expose the scanner itself in Python?  Should the user be using ScannerBuilder?  Should they use NewScan?  Should they use the scanner directly at all or should it be hidden?

Issue Links

• is depended upon by: ARROW-16410 [C++] Scanner -> ScanNode