[BEAM-10760] Cleanup timers lead to unbounded state accumulation in global window

• Type: Bug
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Affects Version/s: 2.21.0
• Fix Version/s: 2.25.0
• Component/s: runner-flink
• Assignee: Thomas Weise
• Reporter: Thomas Weise
• Created: 19/Aug/20 15:34
• Updated: 15/May/21 17:58
• Resolved: 15/May/21 17:58

Description

For each key, the runner sets a cleanup timer that is designed to garbage collect state at the end of a window. For a global window, these timers will stay around until the pipeline terminates. Depending on the key cardinality, this can lead to unbounded state growth, which in the case of the Flink runner is observable in the growth of checkpoint size.

https://lists.apache.org/thread.html/rae268806035688b77646195505e5b7a56568a38feb1e52d6341feedd%40%3Cdev.beam.apache.org%3E

Issue Links

• is duplicated by: BEAM-8212 StatefulDoFnRunner creates GC timers for every record
• links to: GitHub Pull Request #12733
• links to: GitHub Pull Request #12759