[ARROW-4629] [Python] Pandas to arrow conversion slowed down by local imports

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 0.12.1, 0.13.0
• Component/s: Python
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/21165
• Assignee: Florian Jetter
• Reporter: Florian Jetter
• Created: 19/Feb/19 18:12
• Updated: 11/Jan/23 07:35
• Resolved: 20/Feb/19 14:20

Description

The pandas to arrow conversion is currently slowed down significantly by various local import statements.

import pandas as pd
import pyarrow as pa
import cProfile
ser = pd.Series(range(10000))
df = pd.DataFrame({col: ser.copy(deep=True) for col in range(50)})
# Simulate a real dataset, i.e. force copy of data
df = df.astype({col: str for col in range(25)})
prof = cProfile.Profile()

prof.enable()
# a few times to collect statistics
for _ in range(100):
    pa.Table.from_pandas(df, nthreads=1)
prof.disable()
prof.dump_stats("array_conversion.prof")

Issue Links

• links to: GitHub Pull Request #3706