[ARROW-8244] [Python][Parquet] Add `write_to_dataset` option to populate the "file_path" metadata fields

• Type: Wish
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 0.17.0
• Component/s: Python
• Labels: parquet pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/24440
• Assignee: Joris Van den Bossche
• Reporter: Rick Zamora
• Created: 27/Mar/20 14:51
• Updated: 11/Jan/23 07:59
• Resolved: 05/Apr/20 22:21

Description

Prior to [dask#6023|https://github.com/dask/dask/pull/6023], Dask has been using the `write_to_dataset` API to write partitioned parquet datasets.  This PR is switching to a (hopefully temporary) custom solution, because that API makes it difficult to populate the the "file_path"  column-chunk metadata fields that are returned within the optional `metadata_collector` kwarg.  Dask needs to set these fields correctly in order to generate a proper global `"_metadata"` file.

Possible solutions to this problem:

1. Optionally populate the file-path fields within `write_to_dataset`
2. Always populate the file-path fields within `write_to_dataset`
3. Return the file paths for the data written within `write_to_dataset` (up to the user to manually populate the file-path fields)

Issue Links

• links to: GitHub Pull Request #6797