[KAFKA-3938] Fix consumer session timeout issue in Kafka Streams

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.10.1.0
• Component/s: streams
• Labels: architecture
• Assignee: Guozhang Wang
• Reporter: Guozhang Wang
• Created: 08/Jul/16 15:45
• Updated: 03/Oct/16 17:59
• Resolved: 03/Oct/16 17:59

Description

After KIP-62 / KAFKA-3888 is merged in, Kafka Streams should leverage this new feature to fix the session timeout issue that can be caused by:

1. long rebalance period due to state store restoration.
2. exceptional long processing time for a batch of records.

Also we need to consider:

1. state store directory locking mechanism between rebalances while one instance is grabbing tasks from another instance on the same machine.
2. exceptional handling in Kafka Streams: what should we really expose to users and what should be handled automatically?

Issue Links

• Blocked: KAFKA-3888 Allow consumer to send heartbeats in background thread (KIP-62)

Sub-Tasks

• Rethink exception handling in KafkaStreams: Resolved
• Provide a way for KStreams to recover from unclean shutdown: Resolved
• KStream job fails to recover after Kafka broker stopped: Resolved