[SPARK-27037] Pyspark Row .asDict() cannot handle MapType with a Struct as the key or value

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 2.4.0
• Component/s: PySpark
• Assignee: Unassigned
• Reporter: Tanjin Panna
• Created: 04/Mar/19 02:09
• Updated: 12/Dec/22 18:10
• Resolved: 04/Mar/19 03:43

Description

When we have a tuple as the key or value in a MapType and call the .asDict(), we still have a Row in the output for the key and value:

>>> from pyspark.sql import Row 
>>>> df = spark.createDataFrame(
  [ 
    Row(tuple_map={('hello', True): (1234, 111)}), 
    Row(tuple_map={('there', False): (5678, 343)}) 
  ] 
) 

>>> df.schema
StructType(List(StructField(tuple_map,MapType(StructType(List(StructField(_1,StringType,true),StructField(_2,BooleanType,true))),StructType(List(StructField(_1,LongType,true),StructField(_2,LongType,true))),true),true)))

>>> df.show(truncate=False)
+-------------------------------+
|tuple_map                  |
+-------------------------------+
|[[hello, true] -> [1234, 111]] |
|[[there, false] -> [5678, 343]]|
+-------------------------------+

>>> df.head().asDict()
{'tuple_map': {Row(_1=u'hello', _2=True): Row(_1=1234, _2=111)}}