Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-14423

S3: Fetching partition sizes from FS can be expensive when stats are not available in metastore

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Minor
    • Resolution: Fixed
    • None
    • 2.2.0
    • None

    Description

      When partition stats are not available in metastore, it tries to get the file sizes from FS.

      e.g

              at org.apache.hadoop.fs.FileSystem.getContentSummary(FileSystem.java:1487)
              at org.apache.hadoop.hive.ql.stats.StatsUtils.getFileSizeForPartitions(StatsUtils.java:598)
              at org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:235)
              at org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:144)
              at org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:132)
              at org.apache.hadoop.hive.ql.optimizer.stats.annotation.StatsRulesProcFactory$TableScanStatsRule.process(StatsRulesProcFactory.java:126)
              at org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
              at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
      

      This can be quite expensive in some FS like S3. Especially when table is partitioned (e.g TPC-DS store_sales which has 1000s of partitions), query can spend 1000s of seconds just waiting for these information to be pulled in.

      Also, it would be good to remove FS.getContentSummary usage to find out file sizes.

      Attachments

        1. HIVE-14423.1.patch
          5 kB
          Rajesh Balamohan
        2. HIVE-14423.2.patch
          4 kB
          Rajesh Balamohan

        Issue Links

          Activity

            People

              rajesh.balamohan Rajesh Balamohan
              rajesh.balamohan Rajesh Balamohan
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: