[KAFKA-8023] Improve global state store restoration by using multiple update threads

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: streams
• Labels: new-streams-runtime-should-fix performance
• Assignee: Unassigned
• Reporter: Patrik Kleindl
• Created: 01/Mar/19 08:09
• Updated: 04/Oct/23 12:14

Description

Currently global state stores are restored sequentially and the partitions of each global state store are restored sequentially too.

Loop over stores:

https://github.com/apache/kafka/blob/b03e8c234a8aeecd10c2c96b683cfb39b24b548a/streams/src/main/java/org/apache/kafka/streams/processor/internals/GlobalStateManagerImpl.java#L155

Loop over partitions:

https://github.com/apache/kafka/blob/b03e8c234a8aeecd10c2c96b683cfb39b24b548a/streams/src/main/java/org/apache/kafka/streams/processor/internals/GlobalStateManagerImpl.java#L256

It would be a great improvement if one or both of those loops could be processed in parallel.

Possible related task is https://issues.apache.org/jira/browse/KAFKA-6721 

Mail discussion: https://lists.apache.org/thread.html/6fc4772eb8635c04b0ee6682003a99a5ef37ebccffea6c89752e96b1@%3Cusers.kafka.apache.org%3E

Issue Links

• is related to: KAFKA-6721 Consolidate state store management for global stores and normal stores
• relates to: KAFKA-10199 Separate state restoration into separate threads