[SPARK-46721] Some tests failed when cherry-picking https://github.com/apache/spark/pull/43494 back to branch 3.5

• Type: Test
• Status: Resolved
• Priority: Trivial
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: Tests
• Labels: pull-request-available
• Target Version/s: 4.0.0
• Assignee: Bobby Wang
• Reporter: Bobby Wang
• Created: 15/Jan/24 09:03
• Updated: 29/Jan/24 07:01
• Resolved: 29/Jan/24 07:01

Description

When cherry-picking https://github.com/apache/spark/pull/43494 back to branch 3.5 https://github.com/apache/spark/pull/44690, 
I ran into the issue that some tests for Scala 2.12 failed when comparing two maps. It turned out that the function [compareMaps](https://github.com/apache/spark/pull/43494/files#diff-f205431247dd9446f4ce941e5a4620af438c242b9bdff6e7faa7df0194db49acR129) is not so robust for scala 2.12 and scala 2.13.

• scala 2.13

``` scala
Welcome to Scala 2.13.12 (OpenJDK 64-Bit Server VM, Java 17.0.9).
Type in expressions for evaluation. Or try :help.

scala> def compareMaps(lhs: Map[String, Double], rhs: Map[String, Double],
     |                   eps: Double = 0.00000001): Boolean = {
     |     lhs.size == rhs.size &&
     |       lhs.zip(rhs).forall

{ case ((lName, lAmount), (rName, rAmount)) =>      |         lName == rName && (lAmount - rAmount).abs < eps      |       }
     | }
     |   
     | import scala.collection.mutable.HashMap
     | val resources = Map("gpu" > Map("a" -> 1.0, "b" -> 2.0, "c" -> 3.0, "d"> 4.0))
     | val mapped = resources.map { case (rName, addressAmounts) =>      |  rName -> HashMap(addressAmounts.toSeq.sorted: _*)       | }
     | 
     | compareMaps(resources("gpu"), mapped("gpu").toMap)
def compareMaps(lhs: Map[String,Double], rhs: Map[String,Double], eps: Double): Boolean
import scala.collection.mutable.HashMap
val resources: scala.collection.immutable.Map[String,scala.collection.immutable.Map[String,Double]] = Map(gpu -> Map(a -> 1.0, b -> 2.0, c -> 3.0, d -> 4.0))
val mapped: scala.collection.immutable.Map[String,scala.collection.mutable.HashMap[String,Double]] = Map(gpu -> HashMap(a -> 1.0, b -> 2.0, c -> 3.0, d -> 4.0))
val res0: Boolean = true
```

- scala 2.12

``` scala
Welcome to Scala 2.12.14 (OpenJDK 64-Bit Server VM, Java 17.0.9).
Type in expressions for evaluation. Or try :help.

scala> def compareMaps(lhs: Map[String, Double], rhs: Map[String, Double],
     |                   eps: Double = 0.00000001): Boolean = {
     |     lhs.size == rhs.size &&
     |       lhs.zip(rhs).forall { case ((lName, lAmount), (rName, rAmount)) =>      |         lName == rName && (lAmount - rAmount).abs < eps      |       }

     | }
compareMaps: (lhs: Map[String,Double], rhs: Map[String,Double], eps: Double)Boolean

scala> import scala.collection.mutable.HashMap
import scala.collection.mutable.HashMap

scala> val resources = Map("gpu" > Map("a" -> 1.0, "b" -> 2.0, "c" -> 3.0, "d"> 4.0))
resources: scala.collection.immutable.Map[String,scala.collection.immutable.Map[String,Double]] = Map(gpu -> Map(a -> 1.0, b -> 2.0, c -> 3.0, d -> 4.0))

scala> val mapped = resources.map

{ case (rName, addressAmounts) =>      |   rName -> HashMap(addressAmounts.toSeq.sorted: _*)       | }

mapped: scala.collection.immutable.Map[String,scala.collection.mutable.HashMap[String,Double]] = Map(gpu -> Map(b -> 2.0, d -> 4.0, a -> 1.0, c -> 3.0))

scala> compareMaps(resources("gpu"), mapped("gpu").toMap)
res0: Boolean = false
```

The same code bug got different results for Scala 2.12 and Scala 2.13.  This PR tried to rework compareMaps to make tests pass for both scala 2.12 and scala 2.13 

Issue Links

• links to: GitHub Pull Request #44735