[HIVE-21815] Stats in ORC file are parsed twice

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.0.0-alpha-1
• Component/s: ORC
• Assignee: Krisztian Kasa
• Reporter: Gopal Vijayaraghavan
• Created: 31/May/19 15:29
• Updated: 29/Nov/22 10:02
• Resolved: 13/Jun/19 02:23

Description

ORC record reader unnecessarily parses stats twice

      if (orcTail == null) {
        Reader orcReader = OrcFile.createReader(file.getPath(),
            OrcFile.readerOptions(context.conf)
                .filesystem(fs)
                .maxLength(AcidUtils.getLogicalLength(fs, file)));
        orcTail = new OrcTail(orcReader.getFileTail(), orcReader.getSerializedFileFooter(),
            file.getModificationTime());
        if (context.cacheStripeDetails) {
          context.footerCache.put(new FooterCacheKey(fsFileId, file.getPath()), orcTail);
        }
      }
      stripes = orcTail.getStripes();
      stripeStats = orcTail.getStripeStatistics();

We go from Reader -> OrcTail -> StripeStatistics.

stripeStats is read out of the orcTail and is already read inside orcReader.getStripeStatistics().

tez-am-2x-protobuf.svg

Issue Links

• Is contained by: HIVE-26751 Bug Fixes and Improvements for 3.2.0 release
• links to: Review Board