Details
Description
Reported in SPARK-21137. a FileSystem.listStatus call really craws against the local FS, because FileStatus.getPemissions call forces DeprecatedRawLocalFileStatus tp spawn a process to read the real UGI values.
That is: for every other FS, what's a field lookup or even a no-op, on the local FS it's a process exec/spawn, with all the costs. This gets expensive if you have many files.
Attachments
Attachments
Issue Links
- breaks
-
SPARK-21137 Spark reads many small files slowly off local filesystem
- Resolved
- is duplicated by
-
HADOOP-11935 Provide optional native implementation of stat syscall.
- Open
- is related to
-
HADOOP-15128 TestViewFileSystem tests are broken in trunk
- Resolved