[KAFKA-309] Bug in FileMessageSet's append API can corrupt on disk log

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.7
• Component/s: core
• Assignee: Neha Narkhede
• Reporter: Neha Narkhede
• Created: 18/Mar/12 01:14
• Updated: 22/Mar/12 16:59
• Resolved: 22/Mar/12 16:59

Description

In FileMessageSet's append API, we write a ByteBufferMessageSet to a log in the following manner -

while(written < messages.sizeInBytes)
written += messages.writeTo(channel, 0, messages.sizeInBytes)

In ByteBufferMessageSet, the writeTo API uses buffer.duplicate() to append to a channel -

def writeTo(channel: GatheringByteChannel, offset: Long, size: Long): Long =
channel.write(buffer.duplicate)

If the channel doesn't write the ByteBuffer in one call, then we call it again until sizeInBytes bytes are written. But the next call will use buffer.duplicate() to write to the FileChannel, which will write the entire ByteBufferMessageSet again to the file.

Effectively, we have a corrupted set of messages on disk.

Thinking about it, FileChannel is a blocking channel, so ideally, the entire ByteBuffer should be written to the FileChannel in one call. I wrote a test (attached here) and saw that it does. But I'm not aware if there are some corner cases when it doesn't do so. In those cases, Kafka will end up corrupting on disk log segment.

Issue Links

• breaks: KAFKA-308 Corrupted message stored in log segment on disk