[FLINK-12476] [State TTL] Consider setting a default background cleanup strategy in StateTtlConfig

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.8.0
• Fix Version/s: 1.8.0, 1.9.0
• Component/s: Runtime / State Backends
• Labels: pull-request-available
• Release Note: The background cleanup for state with time-to-live can be enabled with the method StateTtlConfig#cleanupInBackground() now. If the state backend supports the background cleanup, the default strategy will kick in.
• Assignee: vinoyang
• Reporter: Andrey Zagrebin
• Created: 10/May/19 07:35
• Updated: 16/Jan/20 15:45
• Resolved: 05/Jun/19 18:55

Description

At the moment we have two efficient background cleanup strategies: incremental for heap and compaction filter for RocksDB. StateTtlConfig has 2 methods to activate them: cleanupIncrementally and cleanupInRocksdbCompactFilter. Each is activated only for certain backend type and inactive for other. They have different tuning parameters.

The idea is to add method cleanupInBackground which would activate default background cleanup. User does not need to think then about details or used backend if not needed. Depending on actually used backend, the corresponding cleanup will kick in. If original strategy is not set with cleanupIncrementally and cleanupInRocksdbCompactFilter then backends should check whether default background cleanup is activated and if so, use it with default parameters.

We can also deprecate the parameterless cleanupInRocksdbCompactFilter() in favour of this new method.

Issue Links

• is depended upon by: FLINK-3089 State API Should Support Data Expiration (State TTL)
• is related to: FLINK-14898 Enable background cleanup of state with TTL by default
• links to: GitHub Pull Request #8459