[SPARK-36003] Implement unary operator `invert` of integral ps.Series/Index

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.2.0
• Component/s: PySpark
• Assignee: Xinrong Meng
• Reporter: Xinrong Meng
• Created: 02/Jul/21 23:14
• Updated: 12/Dec/22 18:11
• Resolved: 12/Jul/21 06:46

Description

 

>>> ~ps.Series([1, 2, 3])
Traceback (most recent call last):
...
pyspark.sql.utils.AnalysisException: cannot resolve '(NOT `0`)' due to data type mismatch: argument 1 requires boolean type, however, '`0`' is of bigint type.;
'Project [unresolvedalias(NOT 0#1L, Some(org.apache.spark.sql.Column$$Lambda$1365/2097273578@53165e1))]
+- Project [__index_level_0__#0L, 0#1L, monotonically_increasing_id() AS __natural_order__#4L]
 +- LogicalRDD [__index_level_0__#0L, 0#1L], false

 

 Currently, unary operator `invert` of integral ps.Series/Index is not supported. We ought to implement that following pandas' behaviors.

Issue Links

• relates to: SPARK-34849 SPIP: Support pandas API layer on PySpark
• links to: [Github] Pull Request #33285 (xinrong-databricks)