[SPARK-37039] np.nan series.astype(bool) should be True

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 4.0.0
• Component/s: PySpark
• Labels: pull-request-available
• Assignee: Haejoon Lee
• Reporter: Yikun Jiang
• Created: 18/Oct/21 06:48
• Updated: 11/Jan/24 08:53
• Resolved: 10/Jan/24 00:55

Description

np.nan series.astype(bool) should be True, rather than Fasle:

https://github.com/apache/spark/blob/46bcef7472edd40c23afd9ac74cffe13c6a608ad/python/pyspark/pandas/data_type_ops/base.py#L147

>>> pd.Series([1, 2, np.nan], dtype=float).astype(bool)
>>> pd.Series([1, 2, np.nan], dtype=str).astype(bool)
>>> pd.Series([datetime.date(1994, 1, 31), datetime.date(1994, 2, 1), np.nan])
0 True
1 True
2 True
dtype: bool

But in pyspark, it is:
0 True
1 True
2 False
dtype: bool

Issue Links

• links to: GitHub Pull Request #44570
• links to: GitHub Pull Request #44684