[SPARK-4495] Memory leak in JobProgressListener due to `spark.ui.retainedJobs` not being used

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.2.0
• Fix Version/s: 1.2.0
• Component/s: Spark Core
• Target Version/s: 1.2.0
• Assignee: Josh Rosen
• Reporter: Josh Rosen
• Created: 19/Nov/14 16:54
• Updated: 20/Nov/14 00:51
• Resolved: 20/Nov/14 00:51

Description

In the JobProgressListener, we never actually remove jobs from the jobIdToData map. In fact, there's a somewhat embarrassing typo: retailed jobs:

  val retailedJobs = conf.getInt("spark.ui.retainedJobs", DEFAULT_RETAINED_JOBS)

This was introduced by my patch for SPARK-2321: I should have added a test that set spark.ui.retainedJobs to some low threshold and checked that we only reported information for that many jobs; there was already a corresponding "test LRU eviction of stages" test that I should have just copied and modified.

This only affects the unreleased 1.2.0 version.

Issue Links

• links to: [Github] Pull Request #3372 (JoshRosen)