[SPARK-27619] MapType should be prohibited in hash expressions

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 2.3.0, 2.3.4, 2.4.0, 2.4.1, 2.4.2, 2.4.3, 2.4.4, 3.0.0
• Fix Version/s: 3.0.0
• Component/s: SQL
• Labels: correctness
• Target Version/s: 3.0.0
• Assignee: Rakesh Raushan
• Reporter: Josh Rosen
• Created: 02/May/19 02:36
• Updated: 02/Mar/20 21:10
• Resolved: 26/Feb/20 18:01

Description

Spark currently allows MapType expressions to be used as input to hash expressions, but I think that this should be prohibited because Spark SQL does not support map equality.

Currently, Spark SQL's map hashcodes are sensitive to the insertion order of map elements:

val a = spark.createDataset(Map(1->1, 2->2) :: Nil)
val b = spark.createDataset(Map(2->2, 1->1) :: Nil)

// Demonstration of how Scala Map equality is unaffected by insertion order:
assert(Map(1->1, 2->2).hashCode() == Map(2->2, 1->1).hashCode())
assert(Map(1->1, 2->2) == Map(2->2, 1->1))
assert(a.first() == b.first())

// In contrast, this will print two different hashcodes:
println(Seq(a, b).map(_.selectExpr("hash(*)").first()))

This behavior might be surprising to Scala developers.

I think there's precedence for banning the use of MapType here because we already prohibit MapType in aggregation / joins / equality comparisons (SPARK-9415) and set operations (SPARK-19893).

If we decide that we want this to be an error then it might also be a good idea to add a spark.sql.legacy flag as an escape-hatch to re-enable the old and buggy behavior (in case applications were relying on it in cases where it just so happens to be safe-by-accident (e.g. maps which only have one entry)).

Alternatively, we could support hashing here if we implemented support for comparable map types (SPARK-18134).

Issue Links

• links to: GitHub Pull Request #27580
• links to: GitHub Pull Request #27719