[SPARK-40238] support scaleUpFactor and initialNumPartition in pyspark rdd API

• Type: Story
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.4.0
• Component/s: PySpark
• Assignee: Unassigned
• Reporter: Ziqi Liu
• Created: 27/Aug/22 00:37
• Updated: 14/Feb/23 08:04

Description

This is a followup on https://issues.apache.org/jira/browse/SPARK-40211

`scaleUpFactor` and `initialNumPartition` config are not supported yet in pyspark rdd take API

 

(see https://github.com/apache/spark/blob/master/python/pyspark/rdd.py#L2799)

basically it hardcoded `scaleUpFactor` as 1 and `initialNumPartition` as 4, therefore pyspark rdd take API is inconsistent with scala API.

 

Anyone familiar with pyspark can help support this (referring to scala implementation)?

Issue Links

• split from: SPARK-40211 Allow executeTake() / collectLimit's number of starting partitions to be customized
• links to: [Github] Pull Request #40009 (wayneguow)