[SPARK-18975] Add an API to remove SparkListener from SparkContext

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 2.2.0
• Component/s: Spark Core
• Assignee: Saisai Shao
• Reporter: Saisai Shao
• Created: 22/Dec/16 07:20
• Updated: 22/Dec/16 19:18
• Resolved: 22/Dec/16 19:18

Description

In current Spark we could add customized SparkListener through SparkContext#addListener API, but there's no API to remove the registered one. In our scenario SparkListener will be added repeatedly accordingly to the changed environment. If lacks the ability to remove listeners, there might be bunch of registered listeners finally, this is unnecessary and potentially affect the performance. So here propose to add an API to remove registered listener.

Issue Links

• is related to: SPARK-18838 High latency of event processing for large jobs
• relates to: SPARK-18838 High latency of event processing for large jobs
• links to: [Github] Pull Request #16382 (jerryshao)