[ARROW-9868] [C++] Provide utility for copying files between filesystems

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Component/s: C++
• Labels: filesystem pull-request-available s3
• External issue URL: https://github.com/apache/arrow/issues/25905
• Assignee: Ben Kietzman
• Reporter: Neal Richardson
• Created: 26/Aug/20 21:38
• Updated: 11/Jan/23 08:09
• Resolved: 11/Sep/20 18:11

Description

CopyStream in arrow/filesystem/util_internal.h does this, but we should expose it, multithread it (can read in one thread while the other thread writes), and further see if there are filesystem-specific optimizations (e.g. S3 multipart uploading/downloading). We may also want a version that takes a FileSelector or vector of paths and parallelizes the operations on them.

Issue Links

• supercedes: ARROW-9909 [C++] Provide a (FileSystem, path) pair to locate files across filesystems
• links to: GitHub Pull Request #8101