[SPARK-37507] Add the TO_BINARY() function

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.3.0
• Component/s: SQL
• Epic Link: New functions in Spark SQL 3.3
• Assignee: Hyukjin Kwon
• Reporter: Max Gekk
• Shepherd: Max Gekk
• Created: 30/Nov/21 13:27
• Updated: 12/Dec/22 18:10
• Resolved: 13/Feb/22 00:33

Description

to_binary(expr, fmt) is a common function available in many other systems to provide a unified entry for string to binary data conversion, where fmt can be utf8, base64, hex and base2 (or whatever the reverse operation to_char()supports).

https://docs.aws.amazon.com/redshift/latest/dg/r_TO_VARBYTE.html

https://docs.snowflake.com/en/sql-reference/functions/to_binary.html

https://cloud.google.com/bigquery/docs/reference/standard-sql/functions-and-operators#format_string_as_bytes

https://docs.teradata.com/r/kmuOwjp1zEYg98JsB8fu_A/etRo5aTAY9n5fUPjxSEynw

Related Spark functions: unbase64, unhex

Issue Links

• links to: [Github] Pull Request #35415 (xinrong-databricks)