[SPARK-43789] Uses 'spark.sql.execution.arrow.maxRecordsPerBatch' in R createDataFrame with Arrow by default

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.5.0
• Fix Version/s: 3.5.0
• Component/s: SparkR
• Assignee: Hyukjin Kwon
• Reporter: Hyukjin Kwon
• Created: 25/May/23 04:44
• Updated: 25/May/23 11:03
• Resolved: 25/May/23 11:03

Description

Now, createDataFrame uses `1` for numPartitions by default, which isn't realistic. Should use larger number for default partitions.

In PySpark, we chunk the input data by 'spark.sql.execution.arrow.maxRecordsPerBatch' size. Should better follow that in SparkR.