[SPARK-36232] Support creating a ps.Series/Index with `Decimal('NaN')` with Arrow disabled

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.3.0
• Component/s: PySpark
• Assignee: Yikun Jiang
• Reporter: Xinrong Meng
• Created: 20/Jul/21 21:19
• Updated: 12/Dec/22 18:11
• Resolved: 16/Oct/21 08:36

Description

 

>>> import decimal as d
>>> import pyspark.pandas as ps
>>> import numpy as np
>>> ps.utils.default_session().conf.set('spark.sql.execution.arrow.pyspark.enabled', True)
>>> ps.Series([d.Decimal(1.0), d.Decimal(2.0), d.Decimal(np.nan)])
0       1
1       2
2    None
dtype: object
>>> ps.utils.default_session().conf.set('spark.sql.execution.arrow.pyspark.enabled', False)
>>> ps.Series([d.Decimal(1.0), d.Decimal(2.0), d.Decimal(np.nan)])
21/07/02 15:01:07 ERROR Executor: Exception in task 6.0 in stage 13.0 (TID 51)
net.razorvine.pickle.PickleException: problem construction object: java.lang.reflect.InvocationTargetException
...

As the code is shown above, we cannot create a Series with `Decimal('NaN')` when Arrow disabled. We ought to fix that.

 

Issue Links

• links to: [Github] Pull Request #34299 (Yikun)
• links to: [Github] Pull Request #34299 (Yikun)