[ARROW-9526] [Python] Memorymapped arrow file conversion to parquet loads everything into RAM

• Type: Bug
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Component/s: C++, Python
• External issue URL: https://github.com/apache/arrow/issues/25594
• Assignee: Unassigned
• Reporter: Sep Dehpour
• Created: 20/Jul/20 06:25
• Updated: 11/Jan/23 08:07

Description

When converting a memory mapped arrow file into parquet file, it loads the whole table into RAM. This effectively negates the point of memory mapping.

If this is not a bug,  perhaps there is a proper way of converting the memorymapped arrow file to parquet without using excessive memory?

 

Example code:

    source = pa.memory_map(path_to_arrow_file, 'r')
    table = pa.ipc.RecordBatchFileReader(source).read_all()
    # The followlng line will load the whole thing into RAM
    pq.write_table(table, path_to_parquet)