[SPARK-30646] transform_keys function throws exception as "Cannot use null as map key", but there isn't any null key in the map

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Invalid
• Affects Version/s: 3.0.0
• Component/s: SQL
• Flags: Patch, Important
• Assignee: Unassigned
• Reporter: Arun Jijo
• Created: 27/Jan/20 06:19
• Updated: 12/Dec/22 18:10
• Resolved: 30/Jan/20 01:37

Description

Have started experimenting Spark 3.0 new SQL functions and along the way found an issue with the transform_keys function. It is raising "Cannot use null as map key" exception but the Map actually doesn't hold any Null values.

Find my spark code below to reproduce the error.

val df = Seq(Map("EID_1"->10000,"EID_2"->25000)).toDF("employees")
df.withColumn("employees",transform_keys($"employees",(k,v)=>lit(k.+("XYX"))))
   .show

Exception in thread "main" java.lang.RuntimeException: Cannot use null as map key.