[SPARK-36274] Fix equality comparison of unordered Categoricals

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.2.0
• Component/s: PySpark
• Assignee: Xinrong Meng
• Reporter: Xinrong Meng
• Created: 23/Jul/21 17:28
• Updated: 24/Jul/21 01:32
• Resolved: 24/Jul/21 01:32

Description

We cannot rely on codes when compare equality of unordered Categoricals.

An example looks like

>>> (ps.Series(pd.Categorical(list('abca'))) == ps.Series(pd.Categorical(list('bcaa'), categories=list('bca')))).sort_index()
0     True
1     True
2     True
3    False
dtype: bool

Issue Links

• links to: [Github] Pull Request #33497 (xinrong-databricks)