In the JobProgressListener, we never actually remove jobs from the jobIdToData map. In fact, there's a somewhat embarrassing typo: retailed jobs:
This was introduced by my patch for
SPARK-2321: I should have added a test that set spark.ui.retainedJobs to some low threshold and checked that we only reported information for that many jobs; there was already a corresponding "test LRU eviction of stages" test that I should have just copied and modified.
This only affects the unreleased 1.2.0 version.