[KAFKA-3776] Unify store and downstream caching in streams

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.10.1.0
• Fix Version/s: 0.10.1.0
• Component/s: streams
• Assignee: Eno Thereska
• Reporter: Eno Thereska
• Reviewer: Eno Thereska
• Created: 02/Jun/16 10:38
• Updated: 02/Oct/16 21:15
• Resolved: 16/Sep/16 16:59

Description

This is an umbrella story for capturing changes to processor caching in Streams as first described in KIP-63. https://cwiki.apache.org/confluence/display/KAFKA/KIP-63%3A+Unify+store+and+downstream+caching+in+streams

Issue Links

• is duplicated by: KAFKA-3101 Optimize Aggregation Outputs
• is related to: KAFKA-3101 Optimize Aggregation Outputs
• links to: GitHub Pull Request #1752
• mentioned in: Page Loading...
• mentioned in: Page Loading...

Sub-Tasks

• Extract the existing LRU cache out of RocksDBStore: Resolved
• Investigate feasibility of caching bytes vs. records: Resolved
• LRU cache should store bytes/object and not records: Resolved
• Avoiding using range queries of RocksDBWindowStore on KStream windowed aggregations: Resolved
• Add new config cache.max.bytes.buffering to the streams configuration: Resolved
• Investigate feasibility of hooking into RocksDb's cache: Resolved
• Forward records in context of cache flushing/eviction: Resolved
• Add cache metrics: Resolved