[SPARK-26068] ChunkedByteBufferInputStream is truncated by empty chunk

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: Spark Core
• Assignee: Liu, Linhong
• Reporter: Liu, Linhong
• Created: 15/Nov/18 05:12
• Updated: 19/Nov/18 14:13
• Resolved: 19/Nov/18 14:11

Description

If ChunkedByteBuffer contains empty chunk in the middle of it, then the ChunkedByteBufferInputStream will be truncated. All data behind the empty chunk will not be read.

The problematic code:

// ChunkedByteBuffer.scala
// Assume chunks.next returns an empty chunk, then we will reach
// else branch no matter chunks.hasNext = true or not. So some data is lost.
override def read(dest: Array[Byte], offset: Int, length: Int): Int = {
  if (currentChunk != null && !currentChunk.hasRemaining && chunks.hasNext)    {
    currentChunk = chunks.next()
  }
  if (currentChunk != null && currentChunk.hasRemaining) {
    val amountToGet = math.min(currentChunk.remaining(), length)
    currentChunk.get(dest, offset, amountToGet)
    amountToGet
  } else {
    close()
    -1
  }
} 

Issue Links

• links to: [Github] Pull Request #23040 (linhong-intel)
• links to: [Github] Pull Request #23040 (linhong-intel)