[FLINK-17998] Support limiting job history size

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 1.12.0
• Component/s: Runtime / Configuration
• Labels: pull-request-available
• Assignee: Daniel Laszlo Magyar
• Reporter: Daniel Laszlo Magyar
• Created: 28/May/20 09:47
• Updated: 12/Feb/21 10:39
• Resolved: 20/Jun/20 13:06

Description

Flink history server stores information for an unlimited number of completed jobs, which is inconvenient to dig through on long-running instances and might cause performance degradation in the long run.

Spark for example addresses this issue by limiting the number of jobs to keep by a configuration parameter spark.history.retainedApplications with a default of 50 jobs to keep.

Introducing a similar configuration option to the Flink history server would improve the usability and performability of long-running history server instances.

Issue Links

• relates to: FLINK-11963 Add time-based cleanup mechanism in history server
• supercedes: FLINK-11963 Add time-based cleanup mechanism in history server
• links to: GitHub Pull Request #12636