[SPARK-43802] unbase64 and unhex codegen are invalid with failOnError

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.1, 3.5.0
• Component/s: SQL
• Assignee: Adam Binford
• Reporter: Adam Binford
• Created: 25/May/23 21:45
• Updated: 27/May/23 02:31
• Resolved: 26/May/23 17:37

Description

to_binary with hex and base64 generate invalid codegen:

spark.range(5).selectExpr('to_binary(base64(cast(id as binary)), "BASE64")').show()

results in

Caused by: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 47, Column 1: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 47, Column 1: Unknown variable or type "BASE64"

because this is the generated code:

/* 107 */         if (!org.apache.spark.sql.catalyst.expressions.UnBase64.isValidBase64(project_value_1))

{ /* 108 */           throw QueryExecutionErrors.invalidInputInConversionError( /* 109 */             ((org.apache.spark.sql.types.BinaryType$) references[1] /* to */), /* 110 */             project_value_1, /* 111 */             BASE64, /* 112 */             "try_to_binary"); /* 113 */         }