[SPARK-1817] RDD zip erroneous when partitions do not divide RDD count

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.9.0, 1.0.0
• Fix Version/s: 1.1.0
• Component/s: Spark Core
• Assignee: Kan Zhang
• Reporter: Michael Malak
• Created: 13/May/14 05:26
• Updated: 04/Jun/14 16:16
• Resolved: 04/Jun/14 05:47

Description

Example:

scala> sc.parallelize(1L to 2L,4).zip(sc.parallelize(11 to 12,4)).collect
res1: Array[(Long, Int)] = Array((2,11))

But more generally, it's whenever the number of partitions does not evenly divide the total number of elements in the RDD.

See https://groups.google.com/forum/#!msg/spark-users/demrmjHFnoc/Ek3ijiXHr2MJ

Issue Links

• depends upon: SPARK-1837 NumericRange should be partitioned in the same way as other sequences