[SAMZA-579] KafkaSystemConsumer drops SSPs on failure

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.9.0
• Fix Version/s: 0.9.0
• Component/s: kafka
• Assignee: Chris Riccomini
• Reporter: Chris Riccomini
• Created: 26/Feb/15 22:05
• Updated: 19/Mar/15 01:07
• Resolved: 28/Feb/15 02:56

Description

While running SAMZA-394, I discovered a bug in KafkaSystemConsumer that causes it to stop consuming under failure scenarios. This does not cause data loss, but can wedge a container until it's restarted.

The trigger appears to be when a BrokerProxy fetches from a broker that's still coming up, and hasn't yet claimed ownership for a TopicAndPartition. When the fetch fails, the BrokerProxy abdicate()s the TopicAndPartition, and KafkaSystemConsumer tries to refresh to get the leader. If there is no leader, the KafkaSystemConsumer drops the SSP. This happens in KafkaSystemConsumer.refreshBrokers.

Issue Links

• is duplicated by: SAMZA-606 KafkaSystemConsumer can stop reading topic-partitions after broker restarts
• relates to: SAMZA-233 BrokerProxies are never started when created during abdication