[BEAM-9794] Flink pipeline with RequiresStableInput fails after Short.MAX_VALUE checkpoints.

• Type: Bug
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Affects Version/s: 2.14.0, 2.15.0, 2.16.0, 2.17.0, 2.18.0, 2.19.0, 2.20.0
• Fix Version/s: 2.21.0
• Component/s: runner-flink
• Assignee: Maximilian Michels
• Reporter: David Morávek
• Created: 21/Apr/20 07:21
• Updated: 16/May/20 14:16
• Resolved: 24/Apr/20 09:26

Description

Full original report: https://lists.apache.org/thread.html/rb2ebfad16d85bcf668978b3defd442feda0903c20db29c323497a672%40%3Cuser.beam.apache.org%3E

The exception comes from: https://github.com/apache/flink/blob/release-1.8.0/flink-runtime/src/main/java/org/apache/flink/runtime/state/OperatorBackendSerializationProxy.java#L68

In the Flink Runner code, each checkpoint results in a new OperatorState (or KeyedState if the stream is keyed):
https://github.com/apache/beam/blob/v2.14.0/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/stableinput/BufferingDoFnRunner.java#L91-L103
https://github.com/apache/beam/blob/v2.14.0/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/stableinput/BufferingDoFnRunner.java#L136-L143

Issue Links

• links to: GitHub Pull Request #11478