Uploaded image for project: 'Tajo'
  1. Tajo
  2. TAJO-1250

RawFileAppender occasionally causes BufferOverflowException

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.0, 0.9.0
    • Fix Version/s: 0.10.0
    • Component/s: Data Shuffle, Storage
    • Labels:
      None

      Description

      The main problem is the maximum bytes size of variant int64.
      if a value is long.max, varint64 length is 10 bytes .

      • Variant int32 : 1~5 bytes
      • Variant int64 : 1~10 bytes
      2014-12-15 13:25:47,908 ERROR org.apache.tajo.engine.planner.physical.HashShuffleFileWriteExec:
      java.nio.BufferOverflowException
              at java.nio.Buffer.nextPutIndex(Buffer.java:513)
              at java.nio.DirectByteBuffer.put(DirectByteBuffer.java:291)
              at org.apache.tajo.storage.RawFile$RawFileAppender.writeRawVarint64(RawFile.java:604)
              at org.apache.tajo.storage.RawFile$RawFileAppender.addTuple(RawFile.java:659)
              at org.apache.tajo.storage.HashShuffleAppender.addTuples(HashShuffleAppender.java:92)
              at org.apache.tajo.engine.planner.physical.HashShuffleFileWriteExec.next(HashShuffleFileWriteExec.java:137)
              at org.apache.tajo.worker.Task.run(Task.java:407)
              at org.apache.tajo.worker.TaskRunner$1.run(TaskRunner.java:276)
              at java.lang.Thread.run(Thread.java:745)
      2014-12-15 13:25:47,910 ERROR org.apache.tajo.worker.Task: java.nio.BufferOverflowException
      java.io.IOException: java.nio.BufferOverflowException
              at org.apache.tajo.engine.planner.physical.HashShuffleFileWriteExec.next(HashShuffleFileWriteExec.java:152)
              at org.apache.tajo.worker.Task.run(Task.java:407)
              at org.apache.tajo.worker.TaskRunner$1.run(TaskRunner.java:276)
              at java.lang.Thread.run(Thread.java:745)
      Caused by: java.nio.BufferOverflowException
              at java.nio.Buffer.nextPutIndex(Buffer.java:513)
              at java.nio.DirectByteBuffer.put(DirectByteBuffer.java:291)
              at org.apache.tajo.storage.RawFile$RawFileAppender.writeRawVarint64(RawFile.java:604)
              at org.apache.tajo.storage.RawFile$RawFileAppender.addTuple(RawFile.java:659)
              at org.apache.tajo.storage.HashShuffleAppender.addTuples(HashShuffleAppender.java:92)
              at org.apache.tajo.engine.planner.physical.HashShuffleFileWriteExec.next(HashShuffleFileWriteExec.java:137)
      

        Attachments

          Activity

            People

            • Assignee:
              jhkim Jinho Kim
              Reporter:
              jhkim Jinho Kim
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: