[TAJO-1250] RawFileAppender occasionally causes BufferOverflowException

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.8.0, 0.9.0
• Fix Version/s: 0.10.0
• Component/s: Data Shuffle, Storage
• Assignee: Jinho Kim
• Reporter: Jinho Kim
• Created: 16/Dec/14 05:52
• Updated: 22/Dec/14 07:46
• Resolved: 22/Dec/14 07:30

Description

The main problem is the maximum bytes size of variant int64.
if a value is long.max, varint64 length is 10 bytes .

• Variant int32 : 1~5 bytes
• Variant int64 : 1~10 bytes

2014-12-15 13:25:47,908 ERROR org.apache.tajo.engine.planner.physical.HashShuffleFileWriteExec:
java.nio.BufferOverflowException
        at java.nio.Buffer.nextPutIndex(Buffer.java:513)
        at java.nio.DirectByteBuffer.put(DirectByteBuffer.java:291)
        at org.apache.tajo.storage.RawFile$RawFileAppender.writeRawVarint64(RawFile.java:604)
        at org.apache.tajo.storage.RawFile$RawFileAppender.addTuple(RawFile.java:659)
        at org.apache.tajo.storage.HashShuffleAppender.addTuples(HashShuffleAppender.java:92)
        at org.apache.tajo.engine.planner.physical.HashShuffleFileWriteExec.next(HashShuffleFileWriteExec.java:137)
        at org.apache.tajo.worker.Task.run(Task.java:407)
        at org.apache.tajo.worker.TaskRunner$1.run(TaskRunner.java:276)
        at java.lang.Thread.run(Thread.java:745)
2014-12-15 13:25:47,910 ERROR org.apache.tajo.worker.Task: java.nio.BufferOverflowException
java.io.IOException: java.nio.BufferOverflowException
        at org.apache.tajo.engine.planner.physical.HashShuffleFileWriteExec.next(HashShuffleFileWriteExec.java:152)
        at org.apache.tajo.worker.Task.run(Task.java:407)
        at org.apache.tajo.worker.TaskRunner$1.run(TaskRunner.java:276)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.nio.BufferOverflowException
        at java.nio.Buffer.nextPutIndex(Buffer.java:513)
        at java.nio.DirectByteBuffer.put(DirectByteBuffer.java:291)
        at org.apache.tajo.storage.RawFile$RawFileAppender.writeRawVarint64(RawFile.java:604)
        at org.apache.tajo.storage.RawFile$RawFileAppender.addTuple(RawFile.java:659)
        at org.apache.tajo.storage.HashShuffleAppender.addTuples(HashShuffleAppender.java:92)
        at org.apache.tajo.engine.planner.physical.HashShuffleFileWriteExec.next(HashShuffleFileWriteExec.java:137)