[ORC-1393] Wrong length of uncompressed stream causes EOFException when reading

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.9.0, 1.8.3
• Assignee: Dmitriy Fingerman
• Reporter: Dmitriy Fingerman
• Created: 09/Mar/23 00:01
• Updated: 29/Mar/23 20:54
• Resolved: 09/Mar/23 20:47

Description

This issue is the root cause of the issue reported in HIVE-27128.

Before 'ORC-516 - Update InStream for column compression', InStream.UncompressedStream class had 'length' field and the length was modifiable in reset() method. 

The reset() method was used in SettableUncompressedStream class in setBuffers() method:

 

public void setBuffers(DiskRangeInfo diskRangeInfo) {
  reset(diskRangeInfo.getDiskRanges(), diskRangeInfo.getTotalLength());
  setOffset(diskRangeInfo.getDiskRanges());
}

After Orc version upgrade in Hive to 1.6.7., and since SettableUncompressedStream class was removed from Orc code base, Hive manages it own copy of SettableUncompressedStream which doesn't pass new length to UncompressedStream when calling reset (because UncompressedStream doesn't accept new length any more in the reset method):

 

public void setBuffers(DiskRangeInfo diskRangeList) {
  reset(diskRangeList.getDiskRanges());
  setOffset(diskRangeList.getDiskRanges());
} 

When investigating the issue reported in HIVE-27128 and comparing the lengths of the InStream.UncompressedStream prior to the upgrade of ORC version in Hive to 1.6.7. (which included ORC-516) and after I noticed that the issue happens with ORC-516 changes because the length of the InStream.UncompressedStream is set once for all row groups, while without those changes the length is dynamic and sometimes is set to bigger value than the initial value.

 

Issue Links

• is required by: HIVE-27128 Exception "Can't finish byte read from uncompressed stream DATA position" when querying ORC table
• links to: GitHub Pull Request #1432