[SPARK-25824] Remove duplicated map entries in `showString`

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Dongjoon Hyun
• Created: 24/Oct/18 18:47
• Updated: 28/Nov/18 15:44
• Resolved: 28/Nov/18 15:44

Description

`showString` doesn't eliminate the duplication. So, it looks different from the result of `collect` and select from saved rows.

Spark 2.2.2

spark-sql> select map(1,2,1,3);
{1:3}

scala> sql("SELECT map(1,2,1,3)").collect
res0: Array[org.apache.spark.sql.Row] = Array([Map(1 -> 3)])

scala> sql("SELECT map(1,2,1,3)").show
+---------------+
|map(1, 2, 1, 3)|
+---------------+
|    Map(1 -> 3)|
+---------------+

Spark 2.3.0 ~ 2.4.0-rc4

spark-sql> select map(1,2,1,3);
{1:3}

scala> sql("SELECT map(1,2,1,3)").collect
res1: Array[org.apache.spark.sql.Row] = Array([Map(1 -> 3)])

scala> sql("CREATE TABLE m AS SELECT map(1,2,1,3) a")
scala> sql("SELECT * FROM m").show
+--------+
|       a|
+--------+
|[1 -> 3]|
+--------+

scala> sql("SELECT map(1,2,1,3)").show
+----------------+
| map(1, 2, 1, 3)|
+----------------+
|[1 -> 2, 1 -> 3]|
+----------------+

Issue Links

• relates to: SPARK-23023 Incorrect results of printing Array/Map/Struct in showString