[ARROW-12315] [R] add max_partitions argument to write_dataset()

• Type: New Feature
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 7.0.0
• Component/s: R
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/28117
• Assignee: Unassigned
• Reporter: Mauricio 'Pachá' Vargas Sepúlveda
• Created: 09/Apr/21 15:42
• Updated: 11/Jan/23 08:25
• Resolved: 10/Nov/21 14:01

Description

the Python docs show that we can pass, say, 1025 partitions
https://arrow.apache.org/docs/_modules/pyarrow/dataset.html

but in R this argument doesn't exist, it would be good to add this for arrow v4.0.0

this is useful, for example, with intl trade datasets:

# d = UN COMTRADE - World's bilateral flows 2019
# 13,050,535 x 22 data.frame
d %>%
          group_by(Year, `Reporter ISO`, `Partner ISO`) %>%
          write_dataset("parquet", hive_style = F)

Error: Invalid: Fragment would be written into 12808 partitions. This exceeds the maximum of 1024

Issue Links

• is duplicated by: ARROW-12363 Write tests for max_partitions argument
• links to: GitHub Pull Request #9972