[KAFKA-6212] Kafka Streams - Incorrect partition rebalancing

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Cannot Reproduce
• Affects Version/s: 0.10.0.1
• Component/s: streams
• Environment: Linux
• Assignee: Unassigned
• Reporter: Ivan Atanasov
• Created: 15/Nov/17 10:50
• Updated: 14/Jan/20 16:31
• Resolved: 14/Jan/20 16:31

Description

Trying to use streaming with version 0.10.0.1 of kafka but it is not working how I'd expect. I realize that this is a fairly old version now but it is what we are running and are not in a position to upgrade right now.

The particular problem I am having is when an extra instance of the streaming app is run using the same application ID. What seems to happen is the newly introduced instance takes half of the partitions available, which is expected but the original instance drops all the partitions it was reading from. therefore from then on, data is only read from half the partitions.

Strangely offsets are still being committed for the other partitions but the data from them is not consumed as expected.

My topology is very simple for now, all it does is a print of the message. Also I have tried making both instances use different client IDs and state directories.

Is this a known bug in 0.10.0.1?

Logs Below:

Instance 1:

[2017-11-15 10:41:41,597] INFO [StreamThread-1] Setting newly assigned partitions [rawEvents-5, rawEvents-6, rawEvents-3, rawEvents-4, rawEvents-9, rawEvents-7, rawEvents-8, rawEvents-1, rawEvents-2, rawEvents-0] for group kafka-stream-test (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
[2017-11-15 10:41:41,616] INFO [StreamThread-1] Creating restoration consumer client for stream task #0_0 (org.apache.kafka.streams.processor.internals.StreamTask)
[2017-11-15 10:41:41,645] INFO [StreamThread-1] Creating restoration consumer client for stream task #0_1 (org.apache.kafka.streams.processor.internals.StreamTask)
[2017-11-15 10:41:41,645] INFO [StreamThread-1] Creating restoration consumer client for stream task #0_2 (org.apache.kafka.streams.processor.internals.StreamTask)
[2017-11-15 10:41:41,646] INFO [StreamThread-1] Creating restoration consumer client for stream task #0_3 (org.apache.kafka.streams.processor.internals.StreamTask)
[2017-11-15 10:41:41,646] INFO [StreamThread-1] Creating restoration consumer client for stream task #0_4 (org.apache.kafka.streams.processor.internals.StreamTask)
[2017-11-15 10:41:41,646] INFO [StreamThread-1] Creating restoration consumer client for stream task #0_5 (org.apache.kafka.streams.processor.internals.StreamTask)
[2017-11-15 10:41:41,647] INFO [StreamThread-1] Creating restoration consumer client for stream task #0_6 (org.apache.kafka.streams.processor.internals.StreamTask)
[2017-11-15 10:41:41,647] INFO [StreamThread-1] Creating restoration consumer client for stream task #0_7 (org.apache.kafka.streams.processor.internals.StreamTask)
[2017-11-15 10:41:41,647] INFO [StreamThread-1] Creating restoration consumer client for stream task #0_8 (org.apache.kafka.streams.processor.internals.StreamTask)
[2017-11-15 10:41:41,648] INFO [StreamThread-1] Creating restoration consumer client for stream task #0_9 (org.apache.kafka.streams.processor.internals.StreamTask)
[2017-11-15 10:42:08,682] INFO [StreamThread-1] Revoking previously assigned partitions [rawEvents-5, rawEvents-6, rawEvents-3, rawEvents-4, rawEvents-9, rawEvents-7, rawEvents-8, rawEvents-1, rawEvents-2, rawEvents-0] for group kafka-stream-test (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
[2017-11-15 10:42:08,682] INFO [StreamThread-1] Removing a task 0_0 (org.apache.kafka.streams.processor.internals.StreamThread)

Instance 2:

[2017-11-15 10:42:08,827] INFO [StreamThread-1] Successfully joined group kafka-stream-test with generation 2 (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2017-11-15 10:42:08,829] INFO [StreamThread-1] Setting newly assigned partitions [rawEvents-5, rawEvents-3, rawEvents-1, rawEvents-2, rawEvents-0] for group kafka-stream-test (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
[2017-11-15 10:42:08,840] INFO [StreamThread-1] Creating restoration consumer client for stream task #0_0 (org.apache.kafka.streams.processor.internals.StreamTask)
[2017-11-15 10:42:08,869] INFO [StreamThread-1] Creating restoration consumer client for stream task #0_1 (org.apache.kafka.streams.processor.internals.StreamTask)
[2017-11-15 10:42:08,870] INFO [StreamThread-1] Creating restoration consumer client for stream task #0_2 (org.apache.kafka.streams.processor.internals.StreamTask)
[2017-11-15 10:42:08,871] INFO [StreamThread-1] Creating restoration consumer client for stream task #0_3 (org.apache.kafka.streams.processor.internals.StreamTask)
[2017-11-15 10:42:08,872] INFO [StreamThread-1] Creating restoration consumer client for stream task #0_5 (org.apache.kafka.streams.processor.internals.StreamTask)