[ORC-1087] Seek overflow in an uncompressed chunk

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.7.0, 1.7.1, 1.7.2
• Fix Version/s: 1.7.3
• Component/s: C++
• Assignee: Quanlong Huang
• Reporter: Quanlong Huang
• Created: 11/Jan/22 03:06
• Updated: 10/Feb/22 08:05
• Resolved: 18/Jan/22 01:58

Description

Reading the attached ORC file with SearchArgument "sr_return_amt > 10000" using the C++ reader will fail with

Corrupt PATCHED_BASE encoded data (pl==0)!

It's ok to read it without the SearchArgument. The java reader is able to read it with the same SearchArgument.

Attached the source codes (scan_with_sarg.cc) for reproducing the issue. Build the ORC lib and compile it by

g++ scan_with_sarg.cc -o scan_with_sarg -I../c++/include -Ic++/include -Lc++/src/ -Lsnappy_ep-prefix/src/snappy_ep-build/ -Llz4_ep-prefix/src/lz4_ep-build/ -Lzlib_ep-prefix/src/zlib_ep-build/ -Lzstd_ep-prefix/src/zstd_ep-build/lib/ -Lprotobuf_ep-prefix/src/protobuf_ep-build/ -lorc -lz -lsnappy -llz4 -lzstd -lprotobuf

Run it as

$ LD_LIBRARY_PATH="$LD_LIBRARY_PATH:zstd_ep-prefix/src/zstd_ep-build/lib/" ./scan_with_sarg 
leaf-0 = (column(id=17) <= 10000), expr = (not leaf-0)
terminate called after throwing an instance of 'orc::ParseError'
  what():  Corrupt PATCHED_BASE encoded data (pl==0)!
Aborted (core dumped)

RCA

The sarg introduces a seek to RowGroup 42. The following codes in DecompressionStream::seek didn't handle the case when uncompressedBufferLength < posInChunk. Then seeks to an illegal position and the length overflow.

if (headerPosition == seekedPosition
    && inputBufferStartPosition <= headerPosition + 3 && inputBufferStart) {
  position.next(); // Skip the input level position.
  size_t posInChunk = position.next(); // Chunk level position.
  // Overflow here! uncompressedBufferLength=30950, posInChunk=39498
  outputBufferLength = uncompressedBufferLength - posInChunk;
  outputBuffer = outputBufferStart + posInChunk;
  return;
}

That chunk is an uncompressed chunk, and the whole chunk is read in pieces. The position (posInChunk) hasn't been read out yet. We need to handle this case.

I think this only happens on uncompressed chunks. For compressed chunks, they are decompressed as a whole. So posInChunk will always be valid in the output buffer.

Issue Links

• is caused by: ORC-614 Implement efficient seek() in decompression streams
• links to: GitHub Pull Request #1008