[HIVE-10161] LLAP: ORC file contains compression buffers larger than bufferSize (OR reader has a bug)

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: llap
• Fix Version/s: llap
• Assignee: Sergey Shelukhin
• Reporter: Gopal Vijayaraghavan
• Created: 31/Mar/15 07:16
• Updated: 30/Apr/20 02:14
• Resolved: 07/Apr/15 01:56

Description

The EncodedReaderImpl will die when reading from the cache, when reading data written by the regular ORC writer

Caused by: java.io.IOException: java.lang.IllegalArgumentException: Buffer size too small. size = 262144 needed = 3919246
        at org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.rethrowErrorIfAny(LlapInputFormat.java:249)
        at org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.nextCvb(LlapInputFormat.java:201)
        at org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.next(LlapInputFormat.java:140)
        at org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.next(LlapInputFormat.java:96)
        at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:350)
        ... 22 more
Caused by: java.lang.IllegalArgumentException: Buffer size too small. size = 262144 needed = 3919246
        at org.apache.hadoop.hive.ql.io.orc.InStream.addOneCompressionBuffer(InStream.java:780)
        at org.apache.hadoop.hive.ql.io.orc.InStream.uncompressStream(InStream.java:628)
        at org.apache.hadoop.hive.ql.io.orc.EncodedReaderImpl.readEncodedColumns(EncodedReaderImpl.java:309)
        at org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:278)
        at org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:48)
        at org.apache.hadoop.hive.common.CallableWithNdc.call(CallableWithNdc.java:37)
        ... 4 more
]], Vertex failed as one or more tasks failed. failedTasks:1, Vertex vertex_1424502260528_1945_1_00 [Map 1] killed/failed due to:null]

Turning off hive.llap.io.enabled makes the error go away.