[ARROW-1796] [Python] RowGroup filtering on file level

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.0.0
• Component/s: C++
• Labels: dataset dataset-parquet-read parquet pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/17793
• Assignee: Joris Van den Bossche
• Reporter: Uwe Korn
• Created: 10/Nov/17 16:28
• Updated: 11/Jan/23 07:17
• Resolved: 11/Jun/20 15:44

Description

We can build upon the API defined in fastparquet for defining RowGroup filters: https://github.com/dask/fastparquet/blob/master/fastparquet/api.py#L296-L300 and translate them into the C++ enums we will define in https://issues.apache.org/jira/browse/PARQUET-1158 . This should enable us to provide the user with a simple predicate pushdown API that we can extend in the background from RowGroup to Page level later on.

Issue Links

• depends upon: ARROW-8039 [Python][Dataset] Support using dataset API in pyarrow.parquet with a minimal ParquetDataset shim
• is related to: PARQUET-1158 [C++] Basic RowGroup filtering
• links to: GitHub Pull Request #2623