[ORC-1065] ReaderImpl.extractFileTail IndexOutOfBoundsException

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.6.12, 1.7.2
• Fix Version/s: 1.7.3, 1.6.13
• Component/s: Java
• Assignee: dzcxzl
• Reporter: dzcxzl
• Created: 28/Dec/21 04:19
• Updated: 21/Jan/22 04:02
• Resolved: 28/Dec/21 19:57

Description

ORC-251 remove `ReaderImpl.extractFileTail`

ORC-685 Add `ReaderImpl.extractFileTail` back

 

In ORC-685, file length is used as readsize, which causes that if the buffer is read from the cache, the use of length is incorrect, resulting in IndexOutOfBoundsException.

long readSize = fileLen != -1? fileLen: buffer.limit();
int psLen = buffer.get((int) (readSize-1)) & 0xff; 

Caused by: java.lang.IndexOutOfBoundsException
    at java.nio.Buffer.checkIndex(Buffer.java:540)
    at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:139)
    at org.apache.orc.impl.ReaderImpl.extractFileTail(ReaderImpl.java:726)
    at org.apache.hadoop.hive.ql.io.orc.LocalCache.getAndValidate(LocalCache.java:103)
    at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$ETLSplitStrategy.getSplits(OrcInputFormat.java:798)
    at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$ETLSplitStrategy.runGetSplitsSync(OrcInputFormat.java:916)
    at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$ETLSplitStrategy.generateSplitWork(OrcInputFormat.java:885)
    at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.scheduleSplits(OrcInputFormat.java:1759)
    at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1703) 

 

Issue Links

• is caused by: ORC-685 Add `ReaderImpl.extractFileTail` back
• links to: GitHub Pull Request #979