[FLINK-9693] Possible memory leak in jobmanager retaining archived checkpoints

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.5.0, 1.6.0
• Fix Version/s: 1.4.3, 1.5.3, 1.6.1, 1.7.0
• Component/s: Runtime / Coordination, Runtime / State Backends
• Labels: pull-request-available

Environment

• Assignee: Till Rohrmann
• Reporter: Steven Zhen Wu
• Created: 29/Jun/18 23:34
• Updated: 28/Feb/19 14:00
• Resolved: 12/Aug/18 04:18

Description

First, some context about the job

• Flink 1.4.1
• stand-alone deployment mode
• embarrassingly parallel: all operators are chained together
• parallelism is over 1,000
• stateless except for Kafka source operators. checkpoint size is 8.4 MB.
• set "state.backend.fs.memory-threshold" so that only jobmanager writes to S3 to checkpoint
• internal checkpoint with 10 checkpoints retained in history

 

Summary of the observations

• 41,567 ExecutionVertex objects retained 9+ GB of memory
• Expanded in one ExecutionVertex. it seems to storing the kafka offsets for source operator

Issue Links

• is superceded by: FLINK-10066 Keep only archived version of previous executions
• links to: GitHub Pull Request #6251
• links to: GitHub Pull Request #6251