[SPARK-35997] Implement comparison operators for CategoricalDtype in pandas API on Spark

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Done
• Affects Version/s: 3.2.0
• Component/s: PySpark
• Assignee: Unassigned
• Reporter: Xinrong Meng
• Created: 02/Jul/21 18:25
• Updated: 27/Jul/21 23:25
• Resolved: 27/Jul/21 23:25

Description

In pandas API on Spark, "<, <=, >, >=" have not been implemented for CategoricalDtype.

We ought to match pandas' behavior.

Issue Links

• relates to: SPARK-34849 SPIP: Support pandas API layer on PySpark

Sub-Tasks

• Implement non-equality comparison operators between two Categoricals: Resolved
• Support comparison between a Categorical and a scalar: Resolved
• Adjust equality comparison operators of Categorical to follow pandas: Resolved
• Fix equality comparison of unordered Categoricals: Resolved