[HADOOP-13427] Eliminate needless uses of FileSystem#{exists(), isFile(), isDirectory()}

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.8.0
• Fix Version/s: 2.9.0, 3.0.0-alpha2
• Component/s: fs
• Hadoop Flags: Reviewed
• Assignee: Mingliang Liu
• Reporter: Steve Loughran
• Created: 26/Jul/16 18:04
• Updated: 09/Sep/20 12:00
• Resolved: 15/Nov/16 19:41

Description

We're cleaning up Hive and Spark's use of FileSystem.exists, because it is often the case we see code of exists+open, exists+delete, when the exists probe is needless. Against object stores, expensive needless.

Hadoop can set an example here by stripping them out. It will also show where there are opportunities to optimise things better and/or improve reporting.

Issue Links

• breaks: HADOOP-17255 JavaKeyStoreProvider fails to create a new key if the keystore is HDFS
• is depended upon by: HADOOP-13525 Optimize uses of FS operations in the ASF analysis frameworks and libraries
• is depended upon by: HADOOP-13321 Deprecate FileSystem APIs that promote inefficient call patterns.
• relates to: TEZ-3540 Eliminate inefficient use of FileSystem API
• relates to: SPARK-16736 remove redundant FileSystem status checks calls from Spark codebase
• relates to: HIVE-14323 Reduce number of FS permissions and redundant FS operations