[ARROW-3428] [Python] from_pandas gives incorrect results when converting floating point to bool

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.12.0
• Component/s: Python
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/19753
• Assignee: Bryan Cutler
• Reporter: Bryan Cutler
• Created: 03/Oct/18 23:30
• Updated: 11/Jan/23 07:27
• Resolved: 10/Jan/19 04:08

Description

When converting Pandas data that contains floating point values to boolean, incorrect results are given

In [2]: import pyarrow as pa
   ...: import pandas as pd
   ...: a = [0.0, 1.0, 2.0, None, float('NaN')]
   ...: 

In [3]: s = pd.Series(a)

In [4]: pa.Array.from_pandas(s, type=pa.bool_())
Out[4]: 
<pyarrow.lib.BooleanArray object at 0x7f1bfd099e68>
[
  False,
  False,
  False,
  False,
  False
]

Expected output should be True when value != 0

This originated from SPARK-25461

Issue Links

• relates to: SPARK-25461 PySpark Pandas UDF outputs incorrect results when input columns contain None
• links to: GitHub Pull Request #2698