[SPARK-36762] Fix Series.isin when Series has NaN values

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0, 3.3.0
• Fix Version/s: 3.2.0
• Component/s: PySpark
• Assignee: dgd_contributor
• Reporter: dgd_contributor
• Created: 15/Sep/21 05:08
• Updated: 18/Sep/21 00:49
• Resolved: 18/Sep/21 00:49

Description

>>> pser = pd.Series([None, 5, None, 3, 2, 1, None, 0, 0])
>>> psser = ps.from_pandas(pser)
>>> pser.isin([1, 3, 5, None])
0    False
1     True
2    False
3     True
4    False
5     True
6    False
7    False
8    False
dtype: bool
>>> psser.isin([1, 3, 5, None])
0    None                                                                       
1    True
2    None
3    True
4    None
5    True
6    None
7    None
8    None
dtype: object

Issue Links

• links to: [Github] Pull Request #34005 (dgd-contributor)