[ARROW-9718] [Python] Make pyarrow.parquet work with the new filesystem interfaces

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Component/s: Python
• Labels: filesystem pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/25773
• Assignee: Joris Van den Bossche
• Reporter: Joris Van den Bossche
• Created: 13/Aug/20 08:46
• Updated: 11/Jan/23 08:08
• Resolved: 07/Sep/20 18:40

Description

The place internally where the "legacy" `pyarrow.filesystem` filesystems are still used is in the pyarrow.parquet module.

It is used in:

• ParquetWriter
• ParquetManifest/ParquetDataset
• write_to_dataset

For ParquetWriter, we need to update this to work with the new filesystems (since ParquetWriter is not dataset related, and thus won't be deprecated).
For ParquetManifest/ParquetDataset, it might not need to be updated, since those might get deprecated itself (to be discussed -> ARROW-9720), and when using the use_legacy_dataset=False option, it already uses the new datasets.
For write_to_dataset, this might depend on how the writing capabilities of the dataset project evolve.

Issue Links

• links to: GitHub Pull Request #7991