[ARROW-8039] [Python][Dataset] Support using dataset API in pyarrow.parquet with a minimal ParquetDataset shim

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.16.0
• Fix Version/s: 0.17.0
• Component/s: C++, Python
• Labels: dataset pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/17077
• Assignee: Joris Van den Bossche
• Reporter: Ben Kietzman
• Created: 09/Mar/20 16:43
• Updated: 11/Jan/23 07:57
• Resolved: 09/Apr/20 13:58

Description

Assemble a minimal ParquetDataset shim backed by pyarrow.dataset.*. Replace the existing ParquetDataset with the shim by default, allow opt-out for users who need the current ParquetDataset

This is mostly exploratory to see which of the python tests fail

Issue Links

• is depended upon by: ARROW-2659 [Python] More graceful reading of empty String columns in ParquetDataset
• is depended upon by: ARROW-2860 [Python][Parquet][C++] Null values in a single partition of Parquet dataset, results in invalid schema on read
• is depended upon by: ARROW-6114 [Python] Datatypes not preserved for partition fields in roundtrip to partitioned parquet dataset
• is depended upon by: ARROW-3861 [Python] ParquetDataset().read columns argument always returns partition column
• is depended upon by: ARROW-5666 [Python] Underscores in partition (string) values are dropped when reading dataset
• is depended upon by: ARROW-5310 [Python] better error message on creating ParquetDataset from empty directory
• is depended upon by: ARROW-5572 [Python] raise error message when passing invalid filter in parquet reading
• is depended upon by: ARROW-2882 [C++][Python] Support AWS Firehose partition_scheme implementation for Parquet datasets
• is depended upon by: ARROW-3388 [C++][Dataset] Automatically detect boolean partition columns
• is depended upon by: ARROW-2366 [Python][C++][Parquet] Support reading Parquet files having a permutation of column order
• is depended upon by: ARROW-3424 [Python] Improved workflow for loading an arbitrary collection of Parquet files
• is depended upon by: ARROW-1796 [Python] RowGroup filtering on file level
• links to: GitHub Pull Request #6303