[ARROW-18225] [Python] write_metadata does not fully use **kwargs

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 11.0.0
• Component/s: Python
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/33411
• Assignee: Miles Granger
• Reporter: François Chareyron
• Created: 02/Nov/22 07:58
• Updated: 11/Jan/23 11:58
• Resolved: 22/Nov/22 07:47

Description

When using write_metadata, kwargs can be used to pass a FileSystem to a ParquetWriter. However, those kwargs are not passed to read_metadata later on despite the function accepting a filesystem argument.

This creates an error when trying to write metadata on a S3FileSystem for example.

def write_metadata(schema, where, metadata_collector=None, **kwargs):
    writer = ParquetWriter(where, schema, **kwargs)
    writer.close()

    if metadata_collector is not None:
        metadata = read_metadata(where) # kwargs should be passed here
        for m in metadata_collector:
            metadata.append_row_groups(m)
        metadata.write_metadata_file(where) # kwargs should be passed here

def read_metadata(where, memory_map=False, decryption_properties=None,
                  filesystem=None):
    ...

Issue Links

• links to: GitHub Pull Request #14574