[ARROW-10426] [C++] Arrow type large_string cannot be written to Parquet type column descriptor

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 3.0.0
• Component/s: C++, R
• Labels: parquet pull-request-available
• Environment: R 4.0.3 on OSX 10.15.7
• External issue URL: https://github.com/apache/arrow/issues/26405
• Assignee: Antoine Pitrou
• Reporter: Gabriel Bassett
• Created: 29/Oct/20 21:49
• Updated: 11/Jan/23 08:13
• Resolved: 23/Nov/20 13:54

Description

When trying to write a dataset in parquet format, arrow errors with the message: "Arrow type large_string cannot be written to Parquet type column descriptor"

arrow::write_dataset(
 dataframe,
 "/directory/",
 "parquet",
 "partitioning" = c("col1", "col2")
)

The dataframe in question is very large with one column containing the text of message board posts encoded in HTML.

Issue Links

• links to: GitHub Pull Request #8632