[ARROW-16616] [Python] Allow lazy evaluation of filters in Dataset and add Datset.filter method

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 11.0.0
• Component/s: Python
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/31969
• Assignee: Alessandro Molina
• Reporter: Alessandro Molina
• Created: 19/May/22 12:59
• Updated: 11/Jan/23 11:45
• Resolved: 12/Dec/22 13:38

Description

To keep the Dataset api compatible with the Table one in terms of analytics capabilities, we should add a Dataset.filter method. The initial POC was based on _table_filter but that required materialising all the Dataset content after filtering as it returned an InMemoryDataset. 

Given that Scanner can filter a dataset without actually materialising the data until a final step happens, it would be good to have Dataset.filter return some form of lazy dataset when the filter is only stored aside and the Scanner is created when data is actually retrieved.

PS: Also update test_dataset_filter test to use the Dataset.filter method

Issue Links

• is related to: ARROW-16409 [C++][Python][R] Deprecate "scanner" (but keep "scan node") from public API
• links to: GitHub Pull Request #13409