[SPARK-48656] ArrayIndexOutOfBoundsException in CartesianRDD getPartitions

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: Spark Core
• Labels: pull-request-available
• Assignee: Wei Guo
• Reporter: Nick Young
• Created: 19/Jun/24 00:12
• Updated: 21/Jun/24 14:28
• Resolved: 21/Jun/24 05:21

Description

```val rdd1 = spark.sparkContext.parallelize(Seq(1, 2, 3), numSlices = 65536)
val rdd2 = spark.sparkContext.parallelize(Seq(1, 2, 3), numSlices = 65536)rdd2.cartesian(rdd1).partitions```

Throws `ArrayIndexOutOfBoundsException: 0` at CartesianRDD.scala:69 because `s1.index * numPartitionsInRdd2 + s2.index` overflows and wraps to 0. We should provide a better error message which indicates the number of partition overflows so it's easier for the user to debug.

Issue Links

• links to: [Github] Pull Request #47019 (wayneguow)
• links to: GitHub Pull Request #47016
• links to: GitHub Pull Request #47019