[SPARK-36000] Support creation and operations of ps.Series/Index with Decimal('NaN')

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Done
• Affects Version/s: 3.2.0
• Fix Version/s: 3.3.0
• Component/s: PySpark
• Assignee: Yikun Jiang
• Reporter: Xinrong Meng
• Created: 02/Jul/21 22:25
• Updated: 17/Nov/21 02:05
• Resolved: 17/Nov/21 02:05

Description

The creation and operations of ps.Series/Index with Decimal('NaN') doesn't work as expected.

That might be due to the underlying PySpark limit.

Please refer to sub-tasks for issues detected.

Issue Links

• relates to: SPARK-34849 SPIP: Support pandas API layer on PySpark

Sub-Tasks

• Support creating a ps.Series/Index with `Decimal('NaN')` with Arrow disabled: Resolved
• hasnans for Series of Decimal(`NaN`): Resolved
• Support arithmetic operations of Series containing Decimal(np.nan): Resolved
• decimal('Nan') is unsupported in net.razorvine.pickle: Resolved