[KAFKA-8922] Failed to get end offsets for topic partitions of global store

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Invalid
• Assignee: Unassigned
• Reporter: Raman Gupta
• Created: 18/Sep/19 17:38
• Updated: 18/Sep/19 20:20
• Resolved: 18/Sep/19 20:20

Description

I have a Kafka stream that fails with this error on startup every time:

org.apache.kafka.streams.errors.StreamsException: Failed to get end offsets for topic partitions of global store test-uiService-dlq-events-table-store after 0 retry attempts. You can increase the number of retries via configuration parameter `retries`.
    at org.apache.kafka.streams.processor.internals.GlobalStateManagerImpl.register(GlobalStateManagerImpl.java:186) ~[kafka-streams-2.3.0.jar:?]
    at org.apache.kafka.streams.processor.internals.AbstractProcessorContext.register(AbstractProcessorContext.java:101) ~[kafka-streams-2.3.0.jar:?]
    at org.apache.kafka.streams.state.internals.RocksDBStore.init(RocksDBStore.java:207) ~[kafka-streams-2.3.0.jar:?]
    at org.apache.kafka.streams.state.internals.KeyValueToTimestampedKeyValueByteStoreAdapter.init(KeyValueToTimestampedKeyValueByteStoreAdapter.java:87) ~[kafka-streams-2.3.0.jar:?]
    at org.apache.kafka.streams.state.internals.WrappedStateStore.init(WrappedStateStore.java:48) ~[kafka-streams-2.3.0.jar:?]
    at org.apache.kafka.streams.state.internals.CachingKeyValueStore.init(CachingKeyValueStore.java:58) ~[kafka-streams-2.3.0.jar:?]
    at org.apache.kafka.streams.state.internals.WrappedStateStore.init(WrappedStateStore.java:48) ~[kafka-streams-2.3.0.jar:?]
    at org.apache.kafka.streams.state.internals.MeteredKeyValueStore.init(MeteredKeyValueStore.java:112) ~[kafka-streams-2.3.0.jar:?]
    at org.apache.kafka.streams.processor.internals.GlobalStateManagerImpl.initialize(GlobalStateManagerImpl.java:123) ~[kafka-streams-2.3.0.jar:?]
    at org.apache.kafka.streams.processor.internals.GlobalStateUpdateTask.initialize(GlobalStateUpdateTask.java:61) ~[kafka-streams-2.3.0.jar:?]
    at org.apache.kafka.streams.processor.internals.GlobalStreamThread$StateConsumer.initialize(GlobalStreamThread.java:229) ~[kafka-streams-2.3.0.jar:?]
    at org.apache.kafka.streams.processor.internals.GlobalStreamThread.initialize(GlobalStreamThread.java:345) ~[kafka-streams-2.3.0.jar:?]
    at org.apache.kafka.streams.processor.internals.GlobalStreamThread.run(GlobalStreamThread.java:270) ~[kafka-streams-2.3.0.jar:?]
Caused by: org.apache.kafka.common.errors.TimeoutException: Failed to get offsets by times in 30001ms

The stream was working fine and then this started happening.

The stream now throws this error on every start. I am now going to attempt to reset the stream and delete its local state.

I hate to say it, but Kafka Streams suck. Its problem after problem.

UPDATE: Some more info: it appears that the brokers may have gotten into some kind of crazy state, for an unknown reason, and now they are just shrinking and expanding ISRs repeatedly. Trying to figure out the root cause of this craziness.