Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
4.0.0, 3.5.2
Description
In https://github.com/apache/spark/pull/47303 , the Base64 function was updated to use `StaticInvoke` instead of directly calling the underlying encode function. This change also updated Base64 of a non-nullable child to be nullable instead of non-nullable. This was backported to Spark 3.5 as well (https://github.com/apache/spark/pull/47325) and has caused issues with failing state schema checks for stateful streaming jobs. This should be fixed to have the nullability based on the child.