[SPARK-25829] remove duplicated map keys with last wins policy

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 3.0.0
• Component/s: SQL
• Assignee: Wenchen Fan
• Reporter: Wenchen Fan
• Created: 25/Oct/18 00:48
• Updated: 18/Feb/20 02:11
• Resolved: 28/Nov/18 15:43

Description

In Spark SQL, we apply "earlier entry wins" semantic to duplicated map keys. e.g.

scala> sql("SELECT map(1,2,1,3)[1]").show
+------------------+
|map(1, 2, 1, 3)[1]|
+------------------+
|                 2|
+------------------+

However, this handling is not applied consistently.

Issue Links

• is duplicated by: SPARK-25823 map_filter can generate incorrect data
• is related to: SPARK-25832 remove newly added map related functions
• links to: [Github] Pull Request #23124 (cloud-fan)
• links to: [Github] Pull Request #23124 (cloud-fan)
• links to: [Github] Pull Request #23217 (mgaido91)
• links to: [Github] Pull Request #23217 (mgaido91)
• links to: GitHub Pull Request #27478

Sub-Tasks

• Remove duplicated map entries in `showString`: Resolved
• should apply "earlier entry wins" in Dataset.collect: Resolved
• should apply "earlier entry wins" in hive map value converter: Resolved