[SPARK-46738] `Cast` displayed different results between Regular Spark and Spark Connect - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Minor
Resolution: Fixed
Affects Version/s: 4.0.0
Fix Version/s: 4.0.0
Component/s: Connect, PySpark
Labels:
- pull-request-available

Description

The following doctest will throw an error in the tests of the pyspark-connect module

Example 5: Decrypt data with key.

>>> import pyspark.sql.functions as sf
>>> df = spark.createDataFrame([(
...     "83F16B2AA704794132802D248E6BFD4E380078182D1544813898AC97E709B28A94",
...     "0000111122223333",)],
...     ["input", "key"]
... )
>>> df.select(sf.try_aes_decrypt(
...     sf.unhex(df.input), df.key
... ).cast("STRING")).show(truncate=False) # doctest: +SKIP
+------------------------------------------------------------------+
|CAST(try_aes_decrypt(unhex(input), key, GCM, DEFAULT, ) AS STRING)|
+------------------------------------------------------------------+
|Spark                                                             |
+------------------------------------------------------------------+

df.select(sf.try_aes_decrypt(
4170        sf.unhex(df.input), df.key
4171    ).cast("STRING")).show(truncate=False)
4172Expected:
4173    +------------------------------------------------------------------+
4174    |CAST(try_aes_decrypt(unhex(input), key, GCM, DEFAULT, ) AS STRING)|
4175    +------------------------------------------------------------------+
4176    |Spark                                                             |
4177    +------------------------------------------------------------------+
4178Got:
4179    +--------------------------------------------------+
4180    |try_aes_decrypt(unhex(input), key, GCM, DEFAULT, )|
4181    +--------------------------------------------------+
4182    |Spark                                             |
4183    +--------------------------------------------------+