[ORC-700] StripePlanner buffer management issues with zero copy

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Unassigned
• Reporter: Panagiotis Garefalakis
• Created: 19/Dec/20 10:20
• Updated: 08/Aug/23 08:56

Description

Observation: Seems that Tests are currently using HadoopShimsPre2_3 (cause of min.hadoop.version) that does not support a zero copy reader – for testing purposes we probably want to test the latest Shims (including the most up-to-date features) – opening a jira for this.

Bumping min.hadoop.version to 2.7.3 and enabling zero copy reveals an underlying issue in buffer management for the StripePlanner:

For example running: testReadFormat_0_11 with conf.setBoolean(OrcConf.USE_ZEROCOPY.getHiveConfName(), true)

java.lang.IllegalArgumentException: tried to release a buffer that was not created by this stream.

	at org.apache.hadoop.fs.FSDataInputStream.releaseBuffer(FSDataInputStream.java:217)
	at org.apache.orc.impl.ZeroCopyShims$ZeroCopyAdapter.releaseBuffer(ZeroCopyShims.java:75)
	at org.apache.orc.impl.RecordReaderUtils$DefaultDataReader.releaseBuffer(RecordReaderUtils.java:134)
	at org.apache.orc.impl.reader.StripePlanner$StreamInformation.releaseBuffers(StripePlanner.java:518)
	at org.apache.orc.impl.reader.StripePlanner.clearStreams(StripePlanner.java:183)
	at org.apache.orc.impl.RecordReaderImpl.clearStreams(RecordReaderImpl.java:1089)
	at org.apache.orc.impl.RecordReaderImpl.beginReadStripe(RecordReaderImpl.java:1124)
	at org.apache.orc.impl.RecordReaderImpl.readStripe(RecordReaderImpl.java:1098)
	at org.apache.orc.impl.RecordReaderImpl.seekToRow(RecordReaderImpl.java:1348)
	at org.apache.orc.TestVectorOrcFile.testReadFormat_0_11(TestVectorOrcFile.java:341)

Issue Links

• is duplicated by: ORC-1473 Zero-copy zeroCopyReadRanges and releaseBuffer bugs