[SPARK-20912] map function with columns as strings

• Type: Improvement
• Status: Resolved
• Priority: Trivial
• Resolution: Won't Fix
• Affects Version/s: 2.3.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Jacek Laskowski
• Created: 29/May/17 07:10
• Updated: 12/Dec/22 18:10
• Resolved: 15/Jun/17 02:20

Description

There's only map function that accepts Column values only. It'd be very helpful to have a variant that accepted String for columns like array or struct.

scala> val kvs = Seq(("key", "value")).toDF("k", "v")
kvs: org.apache.spark.sql.DataFrame = [k: string, v: string]

scala> kvs.printSchema
root
 |-- k: string (nullable = true)
 |-- v: string (nullable = true)


scala> kvs.withColumn("map", map("k", "v")).show
<console>:26: error: type mismatch;
 found   : String("k")
 required: org.apache.spark.sql.Column
       kvs.withColumn("map", map("k", "v")).show
                                 ^
<console>:26: error: type mismatch;
 found   : String("v")
 required: org.apache.spark.sql.Column
       kvs.withColumn("map", map("k", "v")).show
                                      ^

// note $ to create Columns per string
// not very dev-friendly
scala> kvs.withColumn("map", map($"k", $"v")).show
+---+-----+-----------------+
|  k|    v|              map|
+---+-----+-----------------+
|key|value|Map(key -> value)|
+---+-----+-----------------+

Issue Links

• links to: [Github] Pull Request #18144 (liyichao)