[ARROW-18052] [Python] pyarrow.parquet.write_to_dataset permssion denied when write to s3 bucket in another account

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 9.0.0
• Fix Version/s: 11.0.0
• Component/s: Python
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/33254

Language

python

• Assignee: Miles Granger
• Reporter: John
• Created: 14/Oct/22 13:02
• Updated: 11/Jan/23 11:57
• Resolved: 27/Oct/22 15:49

Description

We are using pyarrow.parquet.write_to_dataset to write to an s3 file system in another account.  This fails because write_to_dataset calls pyarrow.dataset.write_dataset with create_dir=True.  There is not a parameter in the call to write_to_dataset that allows you to set create_dir to False.  When setting it to False, this works correctly.

Issue Links

• links to: GitHub Pull Request #14459