[KAFKA-4729] Stores for kstream-kstream join cannot be in-memory

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 0.10.2.0
• Component/s: streams
• Assignee: Unassigned
• Reporter: Eno Thereska
• Created: 03/Feb/17 10:00
• Updated: 05/Sep/19 23:19
• Resolved: 05/Sep/19 23:19

Description

Whereas we can specify in the DSL that stores used for aggregates can be RocksDb-based on in-memory, we cannot do that for stores used for KStream-KStream joins. E.g., the join() methon in KStreamImpl.java creates two state stores and the user does not have the option of having them be in-memory:

StateStoreSupplier thisWindow =
createWindowedStateStore(windows, keySerde, lhsValueSerde, joinThisName + "-store");

StateStoreSupplier otherWindow =
createWindowedStateStore(windows, keySerde, otherValueSerde, joinOtherName + "-store");

Part of the problem is that for joins, stores are not exposed to the user. We might want to rethink that.

Issue Links

• is blocked by: KAFKA-4730 Streams does not have an in-memory windowed store
• Is contained by: KAFKA-8558 KIP-479 - Add StreamJoined Overload to KStream#Join