[HIVE-14323] Reduce number of FS permissions and redundant FS operations

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 2.2.0
• Assignee: Rajesh Balamohan
• Reporter: Rajesh Balamohan
• Created: 25/Jul/16 05:40
• Updated: 13/Feb/18 14:45
• Resolved: 02/Aug/16 00:55

Description

Some examples are given below.
1. When creating stage directory, FileUtils sets the directory permissions by running a set of chgrp and chmod commands. In systems like S3, this would not be relevant.
2. In some cases, fs.delete() is followed by fs.exists(). In this case, it might be redundant to check for exists() (lookup ops are expensive in systems like S3).

Issue Links

• causes: HIVE-18702 INSERT OVERWRITE TABLE doesn't clean the table directory before overwriting
• is depended upon by: HADOOP-13525 Optimize uses of FS operations in the ASF analysis frameworks and libraries
• is depended upon by: HIVE-14269 Performance optimizations for data on S3
• is related to: HADOOP-13321 Deprecate FileSystem APIs that promote inefficient call patterns.
• is related to: HADOOP-13427 Eliminate needless uses of FileSystem#{exists(), isFile(), isDirectory()}
• is related to: SPARK-16736 remove redundant FileSystem status checks calls from Spark codebase
• relates to: HIVE-13925 ETL optimizations