[FLINK-20496] RocksDB partitioned index filter option

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Implemented
• Affects Version/s: 1.10.2, 1.11.2, 1.12.0
• Fix Version/s: 1.13.0
• Component/s: Runtime / State Backends
• Labels: pull-request-available
• Assignee: YufeiLiu
• Reporter: YufeiLiu
• Created: 05/Dec/20 08:05
• Updated: 16/Aug/21 07:30
• Resolved: 04/Mar/21 04:34

Description

When using RocksDBStateBackend and enabling state.backend.rocksdb.memory.managed and state.backend.rocksdb.memory.fixed-per-slot, flink will strictly limited rocksdb memory usage which contains "write buffer" and "block cache". With these options rocksdb stores index and filters in block cache, because in default options index/filters can grows unlimited.

But it's lead another issue, if high-priority cache(configure by state.backend.rocksdb.memory.high-prio-pool-ratio) can't fit all index/filters blocks, it will load all metadata from disk when cache missed, and program went extremely slow. According to Partitioned Index Filters[1], we can enable two-level index having acceptable performance when index/filters cache missed.

Enable these options can get over 10x faster in my case[2], I think we can add an option state.backend.rocksdb.partitioned-index-filters and default value is false, so we can use this feature easily.

[1] Partitioned Index Filters: https://github.com/facebook/rocksdb/wiki/Partitioned-Index-Filters
[2] Deduplicate scenario, state.backend.rocksdb.memory.fixed-per-slot=256M, SSD, elapsed time 4.91ms -> 0.33ms.

Issue Links

• is related to: FLINK-23798 Avoid using reflection to get filter when partition filter is enabled
• links to: GitHub Pull Request #14341