[FLINK-9485] Improving Flink’s timer management for large state

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Runtime / State Backends
• Assignee: Stefan Richter
• Reporter: Stefan Richter
• Created: 01/Jun/18 12:28
• Updated: 09/Sep/19 16:33

Description

See https://docs.google.com/document/d/1XbhJRbig5c5Ftd77d0mKND1bePyTC26Pz04EvxdA7Jc/edit#heading=h.17v0k3363r6q

Issue Links

• is related to: FLINK-14032 Make the cache size of RocksDBPriorityQueueSetFactory configurable
• supercedes: FLINK-9182 async checkpoints for timer service

Sub-Tasks

• Prepare InternalTimerHeap for asynchronous snapshots: Closed
• Introduce TimerState in keyed state backend: Closed
• Generalize/unify state meta info: Closed
• Checkpoint timers as part of managed keyed state instead of raw keyed state: Resolved
• Provide backwards compatibility for timer state of Flink 1.5: Resolved
• Implement timer data structure based on RocksDB: Closed
• Update end-to-end test to use RocksDB backed timers: Resolved
• Ensure serializer upgrades work with timer service remake: Closed
• Tune performance of RocksDB implementation: Closed
• Integrate heap based timers in RocksDB backend with incremental checkpoints: Open
• Add documentation for async/RocksDB-based timers: Closed