[FLINK-32658] State should not be silently removed when ignore-unclaimed-state is false

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.18.0, 1.17.1
• Component/s: Runtime / Checkpointing
• Labels: pull-request-available stale-assigned
• Assignee: Rui Fan
• Reporter: Rui Fan
• Created: 24/Jul/23 11:10
• Updated: 17/Sep/23 22:35

Description

When ignore-unclaimed-state is false and the old state is removed, flink should throw exception. It's similar to removing a stateful operator.

This case occurs not only when the user removes state, but also when the operator is replaced.

For example: upgrade FlinkKafkaConsumer to KafkaSource. All logical are not changed, so the operator id isn't changed. The KafkaSource cannot resume from the state of FlinkKafkaConsumer. However, the new flink job can start, and the state is silently removed in the new job.(The old state is not physically discarded, it is still stored in the state backend, but the new code will never use it.)

It also brings an additional problem: the KafkaSource will snapshot 2 states, it includes the new state of KafkaSource, and the union list state of FlinkKafkaConsumer. Whenever a job resumes from checkpoint, the union List state is inflated. Eventually the state size of kafka offset exceeded 200MB.

Issue Links

• is related to: FLINK-10121 Introduce methods to remove registered operator states
• links to: GitHub Pull Request #23208