[SPARK-46397] sha2(df.a, 1024) throws a different exception in Spark Connect

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: Connect
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: Hyukjin Kwon
• Created: 13/Dec/23 22:23
• Updated: 29/Dec/23 07:16
• Resolved: 29/Dec/23 07:15

Description

from pyspark.sql import functions as sf
spark.range(1).select(sf.sha2(sf.col("id"), 1024)).collect()

Non-connect:

...
pyspark.errors.exceptions.captured.IllegalArgumentException: requirement failed: numBits 1024 is not in the permitted values (0, 224, 256, 384, 512)

Connect:

...
pyspark.errors.exceptions.connect.AnalysisException: [DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE] Cannot resolve "sha2(id, 1024)" due to data type mismatch: Parameter 1 requires the "BINARY" type, however "id" has the type "BIGINT". SQLSTATE: 42K09;
'Project [unresolvedalias(sha2(id#1L, 1024))]
+- Range (0, 1, step=1, splits=Some(1))

Issue Links

• links to: GitHub Pull Request #44529