Details
-
Bug
-
Status: Resolved
-
Minor
-
Resolution: Fixed
-
None
Description
The pandas to arrow conversion is currently slowed down significantly by various local import statements.
import pandas as pd import pyarrow as pa import cProfile ser = pd.Series(range(10000)) df = pd.DataFrame({col: ser.copy(deep=True) for col in range(50)}) # Simulate a real dataset, i.e. force copy of data df = df.astype({col: str for col in range(25)}) prof = cProfile.Profile() prof.enable() # a few times to collect statistics for _ in range(100): pa.Table.from_pandas(df, nthreads=1) prof.disable() prof.dump_stats("array_conversion.prof")
Attachments
Attachments
Issue Links
- links to