[KAFKA-6873] Broker is not returning data including requested offset

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Not A Problem
• Affects Version/s: 1.1.0
• Component/s: consumer
• Environment: ubuntu
• Assignee: Unassigned
• Reporter: Adam Dratwinski
• Created: 07/May/18 11:23
• Updated: 21/May/18 17:10
• Resolved: 21/May/18 17:10

Description

After upgrading Kafka to 1.1.0 from 0.9.x I experience issues related with broker returning incomplete responses. This happens for my all log compacted topics. I am using Golang client (Sarama).

I debugged the issue and found that for some requests brokers return FetchResponse with all messages having offsets lower then requested. For example, I request for offset 1078831, I get FetchResponse with only one message having offset 1078830, which produces missing blocks error. If I request the next offset (1078832), then I get a block with many messages, starting with much higher offset (e.g 1083813). There is a big gap in offsets between these records, probably because I am using log compacted topics, but all expected messages are there.

Sarama client treats this as consumer error:

kafka: response did not contain all the expected topic/partition blocks

For build-in java client this issue is not happening. Looks like it is less restrict regarding the data order, and when the offset is missing in the returned block, it just simply request the next offset.

I reported this issue at Shopify/sarama Github project (see https://github.com/Shopify/sarama/issues/1087), where I got response, that this seems to be Kafka bug, as according to the documentation, in this situation broker should never return only messages having lower offsets.