Description
Calling repartition on a PySpark RDD to increase the number of partitions results in highly skewed partition sizes, with most having 0 rows. The repartition method should evenly spread out the rows across the partitions, and this behavior is correctly seen on the Scala side.
Please reference the following code for a reproducible example of this issue:
# Python
num_partitions = 20000
a = sc.parallelize(range(int(1e6)), 2) # start with 2 even partitions
l = a.repartition(num_partitions).glom().map(len).collect() # get length of each partition
min(l), max(l), sum(l)/len(l), len(l) # skewed!
# Scala
val numPartitions = 20000
val a = sc.parallelize(0 until 1e6.toInt, 2) # start with 2 even partitions
val l = a.repartition(numPartitions).glom().map(_.length).collect() # get length of each partition
print(l.min, l.max, l.sum/l.length, l.length) # even!
The issue here is that highly skewed partitions can result in severe memory pressure in subsequent steps of a processing pipeline, resulting in OOM errors.