[KAFKA-4587] Rethink Unification of Caching with Dedupping

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: streams
• Assignee: Unassigned
• Reporter: Guozhang Wang
• Created: 04/Jan/17 06:54
• Updated: 04/Aug/20 19:49

Description

This is discussed in PR https://github.com/apache/kafka/pull/1588

In order to support user-customizable state store suppliers in the DSL we did the following:

1) Introduce a TupleForwarder to forward tuples from cached stores that is wrapping user customized stores.
2) Narrow the scope to only dedup on forwarding if it is the default CachingXXStore with wrapper RocksDB.

With this, the unification of dedupping and caching is less useful now, and we are complicating the inner implementations for forwarding a lot. We need to re-consider this feature with finer granularity of turning on / off caching per store, potentially with explicit triggers.

Issue Links

• is related to: KAFKA-7224 KIP-328: Add spill-to-disk for Suppression
• is related to: KAFKA-7222 KIP-328: Add Window Grace Period (and deprecate Window Retention)
• is related to: KAFKA-7223 KIP-328: Add in-memory Suppression
• mentioned in: Page Loading...