Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-27619

MapType should be prohibited in hash expressions

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.4.0
    • Fix Version/s: None
    • Component/s: SQL
    • Labels:

      Description

      Spark currently allows MapType expressions to be used as input to hash expressions, but I think that this should be prohibited because Spark SQL does not support map equality.

      Currently, Spark SQL's map hashcodes are sensitive to the insertion order of map elements:

      val a = spark.createDataset(Map(1->1, 2->2) :: Nil)
      val b = spark.createDataset(Map(2->2, 1->1) :: Nil)
      
      // Demonstration of how Scala Map equality is unaffected by insertion order:
      assert(Map(1->1, 2->2).hashCode() == Map(2->2, 1->1).hashCode())
      assert(Map(1->1, 2->2) == Map(2->2, 1->1))
      assert(a.first() == b.first())
      
      // In contrast, this will print two different hashcodes:
      println(Seq(a, b).map(_.selectExpr("hash(*)").first()))

      This behavior might be surprising to Scala developers.

      I think there's precedence for banning the use of MapType here because we already prohibit MapType in aggregation / joins / equality comparisons (SPARK-9415) and set operations (SPARK-19893).

      If we decide that we want this to be an error then it might also be a good idea to add a spark.sql.legacy flag as an escape-hatch to re-enable the old and buggy behavior (in case applications were relying on it in cases where it just so happens to be safe-by-accident (e.g. maps which only have one entry)).

      Alternatively, we could support hashing here if we implemented support for comparable map types (SPARK-18134).

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              joshrosen Josh Rosen
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated: