[HUDI-1812] Add explicit index state TTL option for Flink writer

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Done
• Fix Version/s: 0.9.0
• Component/s: flink
• Labels: pull-request-available
• Assignee: 谢波
• Reporter: Danny Chen
• Created: 19/Apr/21 10:19
• Updated: 21/Apr/21 12:14
• Resolved: 21/Apr/21 12:14

Description

Add option:

public static final ConfigOption<Double> INDEX_STATE_TTL = ConfigOptions
    .key("index.state.ttl")
    .doubleType()
    .defaultValue(1.5D)
    .withDescription("index state ttl in days. default is 1.5 day.");

If the state expires but there are still updates for old records, the records would be recognized as INSERT instead of UPDATE thus some data duplication.

Issue Links

• links to: GitHub Pull Request #2853