[SPARK-49476] Base64 nullability change

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0, 3.5.2
• Fix Version/s: 4.0.0, 3.5.3
• Component/s: SQL
• Labels: pull-request-available
• Assignee: Adam Binford
• Reporter: Adam Binford
• Created: 30/Aug/24 20:10
• Updated: 03/Sep/24 21:36
• Resolved: 02/Sep/24 06:06

Description

In https://github.com/apache/spark/pull/47303 , the Base64 function was updated to use `StaticInvoke` instead of directly calling the underlying encode function. This change also updated Base64 of a non-nullable child to be nullable instead of non-nullable. This was backported to Spark 3.5 as well (https://github.com/apache/spark/pull/47325) and has caused issues with failing state schema checks for stateful streaming jobs. This should be fixed to have the nullability based on the child.

Issue Links

• links to: GitHub Pull Request #47941
• links to: GitHub Pull Request #47964