[HIVE-14165] Remove Hive file listing during split computation

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.1.0
• Fix Version/s: 4.0.0-alpha-1
• Labels: pull-request-available
• Target Version/s: 3.0.0
• Assignee: Peter Varga
• Reporter: Abdullah Yousufi
• Created: 05/Jul/16 22:44
• Updated: 17/Nov/22 08:55
• Resolved: 19/Jan/21 09:52

Description

The Hive side listing in FetchOperator.java is unnecessary, since Hadoop's FileInputFormat.java will list the files during split computation anyway to determine their size. One way to remove this is to catch the InvalidInputFormat exception thrown by FileInputFormat#getSplits() on the Hive side instead of doing the file listing beforehand.

For S3 select queries on partitioned tables, this results in a 2x speedup.

Issue Links

• is depended upon by: HADOOP-13525 Optimize uses of FS operations in the ASF analysis frameworks and libraries
• is related to: MAPREDUCE-6760 LocatedFileStatusFetcher to use listFiles(recursive)
• relates to: HADOOP-13208 S3A listFiles(recursive=true) to do a bulk listObjects instead of walking the pseudo-tree of directories
• links to: GitHub Pull Request #1866
• links to: Review Board