[ARROW-14701] [Python] parquet.write_table has an undocumented and silent cap on row_group_size

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 6.0.0
• Fix Version/s: 7.0.0
• Component/s: Python
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/30241
• Assignee: Will Jones
• Reporter: Will Jones
• Created: 12/Nov/21 17:03
• Updated: 11/Jan/23 08:41
• Resolved: 10/Dec/21 21:57

Description

from io import BytesIO
import pandas as pd
import pyarrow
from pyarrow import parquet
from pyarrow import fs

print(pyarrow._version_)

def check_row_groups_created(size: int):
    df = pd.DataFrame({"a": range(size)})
    t = pyarrow.Table.from_pandas(df)
    buffer = BytesIO()
    parquet.write_table(t, buffer, row_group_size=size)
    buffer.seek(0)
    print(parquet.read_metadata(buffer))
    
check_row_groups_created(50_000_000)
check_row_groups_created(100_000_000) 

outputs:

6.0.0

<pyarrow._parquet.FileMetaData object at 0x7f838584ab80>
created_by: parquet-cpp-arrow version 6.0.0
num_columns: 1
num_rows: 50000000
num_row_groups: 1
format_version: 1.0
serialized_size: 1493

<pyarrow._parquet.FileMetaData object at 0x7f838584ab80>
created_by: parquet-cpp-arrow version 6.0.0
num_columns: 1
num_rows: 100000000
num_row_groups: 2
format_version: 1.0 
serialized_size: 1640 

Issue Links

• links to: GitHub Pull Request #11928