[SPARK-5967] JobProgressListener.stageIdToActiveJobIds never cleared

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.2.0, 1.2.1, 1.3.0
• Fix Version/s: 1.2.2, 1.3.0
• Component/s: DStreams, Web UI
• Target Version/s: 1.2.2, 1.3.0
• Assignee: Tathagata Das
• Reporter: Tathagata Das
• Created: 24/Feb/15 06:48
• Updated: 17/Apr/15 18:39
• Resolved: 24/Feb/15 19:03

Description

The hashmap stageIdToActiveJobIds in JobProgressListener is never cleared. So the hashmap keep increasing in size indefinitely. This is a blocker for 24/7 running applications like Spark Streaming apps.

Issue Links

• links to: [Github] Pull Request #4741 (tdas)