[SPARK-47307] Spark 3.3 produces invalid base64

• Type: Bug
• Status: Open
• Priority: Blocker
• Resolution: Unresolved
• Affects Version/s: 3.3.0
• Component/s: SQL
• Labels: correctness pull-request-available
• Assignee: Unassigned
• Reporter: Willi Raschkowski
• Created: 06/Mar/24 17:23
• Updated: 20/Mar/24 06:22

Description

SPARK-37820 was introduced in Spark 3.3 and breaks behavior of base64 (which is fine but shouldn't happen between minor version).

Spark 3.2

>>> spark.sql(f"""SELECT base64('{'a' * 58}') AS base64""").collect()[0][0]
'YWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYQ=='

Note the different output in Spark 3.3 (the addition of \r\n newlines).

Spark 3.3

>>> spark.sql(f"""SELECT base64('{'a' * 58}') AS base64""").collect()[0][0]
'YWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFh\r\nYQ=='

The former decodes fine with the base64 on my machine but the latter does not:

$ pbpaste | base64 --decode
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa%

$ pbpaste | base64 --decode
base64: stdin: (null): error decoding base64 input stream

Issue Links

• links to: GitHub Pull Request #45408