[ARROW-11567] [C++][Compute] Variance kernel has precision issue

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 4.0.0
• Component/s: C++
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/18508
• Assignee: Yibo Cai
• Reporter: Yibo Cai
• Created: 09/Feb/21 04:33
• Updated: 11/Jan/23 08:20
• Resolved: 02/Mar/21 13:07

Description

For long arrays, variance kernel is not accurate compared with numpy. See below example.

In [1]: import numpy as np

In [2]: import pyarrow.compute as pc

In [3]: t = np.arange(321000, dtype='float')

In [4]: np.var(t)
Out[4]: 8586749999.916667

In [5]: pc.variance(t)
Out[5]: <pyarrow.DoubleScalar: 8586749999.899047>

Issue Links

• is related to: ARROW-11758 [C++][Compute] Summation kernel round-off error
• links to: GitHub Pull Request #9604