[HADOOP-14600] LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.7.3
• Fix Version/s: 3.1.0
• Component/s: fs

Environment

file:// in a dir with many files

• Target Version/s: 3.1.0
• Hadoop Flags: Reviewed
• Assignee: Ping Liu
• Reporter: Steve Loughran
• Created: 27/Jun/17 20:42
• Updated: 28/Apr/19 12:16
• Resolved: 02/Dec/17 06:24

Description

Reported in SPARK-21137. a FileSystem.listStatus call really craws against the local FS, because FileStatus.getPemissions call forces DeprecatedRawLocalFileStatus tp spawn a process to read the real UGI values.

That is: for every other FS, what's a field lookup or even a no-op, on the local FS it's a process exec/spawn, with all the costs. This gets expensive if you have many files.

Issue Links

• breaks: SPARK-21137 Spark reads many small files slowly off local filesystem
• is duplicated by: HADOOP-11935 Provide optional native implementation of stat syscall.
• is related to: HADOOP-15128 TestViewFileSystem tests are broken in trunk