[SPARK-36143] Adjust `astype` of fractional Series with missing values to follow pandas

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.2.0
• Component/s: PySpark
• Assignee: Xinrong Meng
• Reporter: Xinrong Meng
• Created: 14/Jul/21 23:50
• Updated: 28/Jul/21 18:27
• Resolved: 28/Jul/21 18:27

Description

>>> pser = pd.Series([1, 2, np.nan], dtype=float)
>>> psser = ps.from_pandas(pser)
>>> pser.astype(int)
...
 ValueError: Cannot convert non-finite values (NA or inf) to integer
>>> psser.astype(int)
 0 1.0
 1 2.0
 2 NaN
 dtype: float64

As shown above, astype of Series with fractional missing values doesn't behave the same as pandas, we ought to adjust that.

Issue Links

• links to: [Github] Pull Request #33466 (xinrong-databricks)