[ARROW-9623] [Python] Performance difference between pc.multiply vs pd.multiply

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 1.0.0
• Component/s: Python
• Environment: Windows ; Pyarrow 1.0.0
• External issue URL: https://github.com/apache/arrow/issues/25684
• Assignee: Unassigned
• Reporter: H G
• Created: 03/Aug/20 09:44
• Updated: 11/Jan/23 08:07

Description

Wanted to report the performance difference observed between Pandas and Pyarrow.

 

 import numpy as np
 import pandas as pd
 import pyarrow as pa
 import pyarrow.compute as pc
df = pd.DataFrame(np.random.randn(100000000))
 %timeit -n 5 -r 5 df.multiply(df)
table = pa.Table.from_pandas(df)
 %timeit -n 5 -r 5 pc.multiply(table[0],table[0])

Results:

%timeit -n 5 -r 5 df.multiply(df)
 374 ms ± 15.9 ms per loop (mean ± std. dev. of 5 runs, 5 loops each)``

 

%timeit -n 5 -r 5 pc.multiply(table[0],table[0])
 698 ms ± 297 ms per loop (mean ± std. dev. of 5 runs, 5 loops each)