[SPARK-18134] SQL: MapType in Group BY and Joins not working

• Type: Bug
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.5.0, 1.5.1, 1.5.2, 1.6.0, 1.6.1, 1.6.2, 2.0.0, 2.0.1, 2.1.0
• Component/s: SQL
• Target Version/s: 3.0.0
• Assignee: Unassigned
• Reporter: Christian Zorneck
• Created: 27/Oct/16 07:29
• Updated: 14/May/21 15:39

Description

Since version 1.5 and issue SPARK-9415, MapTypes can no longer be used in GROUP BY and join clauses. This makes it incompatible to HiveQL. So, a Hive feature was removed from Spark. This makes Spark incompatible to various HiveQL statements.

Issue Links

• is related to: SPARK-34819 MapType supports orderable semantics
• relates to: SPARK-19893 should not run DataFrame set oprations with map type
• links to: [Github] Pull Request #15970 (hvanhovell)
• links to: [Github] Pull Request #19330 (jinxing64)
• links to: [Github] Pull Request #32552 (maropu)
• links to: [Github] Pull Request #32552 (maropu)
• links to: GitHub Pull Request #15970

Sub-Tasks

• Add sorting functionality in MapType: Resolved
• Add a new expression "SortMaps" to sort the entries inside a "MapData": Resolved