[HIVE-14423] S3: Fetching partition sizes from FS can be expensive when stats are not available in metastore

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 2.2.0
• Labels: TODOC2.2
• Target Version/s: 2.1.0
• Assignee: Rajesh Balamohan
• Reporter: Rajesh Balamohan
• Created: 04/Aug/16 08:18
• Updated: 26/Jul/17 03:30
• Resolved: 06/Aug/16 01:00

Description

When partition stats are not available in metastore, it tries to get the file sizes from FS.

e.g

        at org.apache.hadoop.fs.FileSystem.getContentSummary(FileSystem.java:1487)
        at org.apache.hadoop.hive.ql.stats.StatsUtils.getFileSizeForPartitions(StatsUtils.java:598)
        at org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:235)
        at org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:144)
        at org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:132)
        at org.apache.hadoop.hive.ql.optimizer.stats.annotation.StatsRulesProcFactory$TableScanStatsRule.process(StatsRulesProcFactory.java:126)
        at org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
        at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)

This can be quite expensive in some FS like S3. Especially when table is partitioned (e.g TPC-DS store_sales which has 1000s of partitions), query can spend 1000s of seconds just waiting for these information to be pulled in.

Also, it would be good to remove FS.getContentSummary usage to find out file sizes.

Issue Links

• is depended upon by: HADOOP-13525 Optimize uses of FS operations in the ASF analysis frameworks and libraries
• is related to: HIVE-13925 ETL optimizations
• relates to: HIVE-13901 Hivemetastore add partitions can be slow depending on filesystems