[HADOOP-13321] Deprecate FileSystem APIs that promote inefficient call patterns.

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0-alpha4
• Fix Version/s: 3.0.0-alpha4
• Component/s: fs
• Target Version/s: 3.0.0-alpha4
• Hadoop Flags: Reviewed
• Assignee: Mingliang Liu
• Reporter: Chris Nauroth
• Created: 24/Jun/16 23:59
• Updated: 25/Feb/17 02:48
• Resolved: 23/Feb/17 21:33

Description

FileSystem contains several methods that act as convenience wrappers over calling getFileStatus and retrieving a single property of the returned FileStatus. These methods have a habit of fostering inefficient call patterns in applications, resulting in multiple redundant getFileStatus calls. For HDFS, this translates into wasteful NameNode RPC traffic. For file systems backed by cloud object stores, this translates into wasteful HTTP traffic. This issue proposes to deprecate these methods and instead encourage applications to call getFileStatus and then reuse the same FileStatus instance as needed.

Issue Links

• depends upon: HADOOP-13427 Eliminate needless uses of FileSystem#{exists(), isFile(), isDirectory()}
• is depended upon by: HADOOP-13525 Optimize uses of FS operations in the ASF analysis frameworks and libraries
• is related to: SPARK-16736 remove redundant FileSystem status checks calls from Spark codebase
• relates to: HADOOP-12876 [Azure Data Lake] Support for process level FileStatus cache to optimize GetFileStatus frequent operations
• relates to: HADOOP-13204 Über-jira: S3a phase III: scale and tuning
• relates to: HIVE-10223 Consolidate several redundant FileSystem API calls.
• relates to: HIVE-14323 Reduce number of FS permissions and redundant FS operations
• relates to: PIG-4442 Eliminate redundant RPC call to get file information in HPath.