[FLINK-10471] State TTL cleanup using RocksDb compaction filter

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Implemented
• Affects Version/s: 1.7.0
• Fix Version/s: 1.8.0
• Component/s: Runtime / State Backends
• Labels: pull-request-available

Release Note

The used build of FRocksdb is based on the upgraded version 5.17.2 of Rocksdb. In case of Mac OS X, the Rocksdb version 5.17.2 is supported only for OS X version >= 10.13.
See also: https://github.com/facebook/rocksdb/issues/4862

• Assignee: Andrey Zagrebin
• Reporter: Andrey Zagrebin
• Created: 01/Oct/18 12:21
• Updated: 19/Mar/20 20:31
• Resolved: 21/Feb/19 12:27

Description

In case of rocksdb backend, we can piggy back compaction using custom compaction filter which checks our last access timestamp and discards expired values. It requires contributing a C++ Flink TTL specific filter to Rocksdb, like for cassandra. At the moment RocksDB does not support compaction filter plugins (see PR discussion), it is under development.

Meanwhile, we can apply to strategies to enable this feature in Flink:

• Release and maintain a temporary fork of RocksDB for Flink: FRocksDB and merge TTL filter into this fork (used in Flink 1.8)
• Build C++ TTL filter separately, pack this C++ lib into its JNI java client jar and load it in Flink additionally to vanila RocksDB (Flink RocksDB extensions, planned for Flink 1.9)

The second strategy is more flexible in the long run.

Issue Links

• is depended upon by: FLINK-3089 State API Should Support Data Expiration (State TTL)
• is related to: FLINK-16686 [State TTL] Make user class loader available in native RocksDB compaction thread
• is related to: FLINK-15506 State TTL Compaction Filter option off by default
• links to: GitHub Pull Request #7163