[ARROW-9633] [C++] Do not toggle memory mapping globally in LocalFileSystem

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: C++
• External issue URL: https://github.com/apache/arrow/issues/25694
• Assignee: Unassigned
• Reporter: Wes McKinney
• Created: 03/Aug/20 16:23
• Updated: 11/Jan/23 08:08

Description

In the context of the Datasets API, some file formats benefit greatly from memory mapping (like Arrow IPC files) while other less so. Additionally, in some scenarios, memory mapping could fail when used on network-attached storage devices. Since a filesystem may be used to read different kinds of files and use both memory mapping and non-memory mapping, and additionally the Datasets API should be able to fall back on non-memory mapping if the attempt to memory map fails, it would make sense to have a non-global option for this:

https://github.com/apache/arrow/blob/master/cpp/src/arrow/filesystem/localfs.h

I would suggest adding a new filesystem API with something like OpenMappedInputFile with some options to control the behavior when memory mapping is not possible. These options may be among:

• Falling back on a normal RandomAccessFile
• Reading the entire file into memory (or even tmpfs?) and then wrapping it in a BufferReader
• Failing