Uploaded image for project: 'ORC'
  1. ORC
  2. ORC-1393

Wrong length of uncompressed stream causes EOFException when reading

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 1.9.0, 1.8.3
    • None
    • None

    Description

      This issue is the root cause of the issue reported in HIVE-27128.

      Before 'ORC-516 - Update InStream for column compression', InStream.UncompressedStream class had 'length' field and the length was modifiable in reset() method. 

      The reset() method was used in SettableUncompressedStream class in setBuffers() method:

       

      public void setBuffers(DiskRangeInfo diskRangeInfo) {
        reset(diskRangeInfo.getDiskRanges(), diskRangeInfo.getTotalLength());
        setOffset(diskRangeInfo.getDiskRanges());
      }

      After Orc version upgrade in Hive to 1.6.7., and since SettableUncompressedStream class was removed from Orc code base, Hive manages it own copy of SettableUncompressedStream which doesn't pass new length to UncompressedStream when calling reset (because UncompressedStream doesn't accept new length any more in the reset method):

       

      public void setBuffers(DiskRangeInfo diskRangeList) {
        reset(diskRangeList.getDiskRanges());
        setOffset(diskRangeList.getDiskRanges());
      } 

      When investigating the issue reported in HIVE-27128 and comparing the lengths of the InStream.UncompressedStream prior to the upgrade of ORC version in Hive to 1.6.7. (which included ORC-516) and after I noticed that the issue happens with ORC-516 changes because the length of the InStream.UncompressedStream is set once for all row groups, while without those changes the length is dynamic and sometimes is set to bigger value than the initial value.

       

      Attachments

        Issue Links

          Activity

            People

              difin Dmitriy Fingerman
              difin Dmitriy Fingerman
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: