[ARROW-1311] python hangs after write a few parquet tables

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 0.5.0
• Fix Version/s: 0.6.0
• Component/s: Python
• Environment: Python 3.5.2, pyarrow 0.5.0
• External issue URL: https://github.com/apache/arrow/issues/17324
• Assignee: Wes McKinney
• Reporter: Keith Curtis
• Created: 01/Aug/17 22:37
• Updated: 11/Jan/23 07:13
• Resolved: 02/Aug/17 21:59

Description

I had a program to read some csv files (a few million rows each, 9 columns), and converted with:

import os
import pandas as pd

import pyarrow.parquet as pq
import pyarrow

def to_parquet(output_file, csv_file):
    df = pd.read_csv(csv_file)
    df['gecco_variant'] = [ v.lstrip('0') for v in df['gecco_variant']]
    table = pyarrow.Table.from_pandas(df)
    pq.write_table(table, output_file)

The first csv file would always complete, but python would hang on the second or third file, and sometimes on a much later file.