[SPARK-9806] Don't share ReplayListenerBus between multiple applications

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.4.0
• Fix Version/s: 1.4.2, 1.5.0
• Component/s: Web UI
• Target Version/s: 1.4.2, 1.5.0
• Assignee: Rohit Agarwal
• Reporter: Rohit Agarwal
• Created: 11/Aug/15 00:44
• Updated: 12/Aug/15 06:21
• Resolved: 12/Aug/15 06:21

Description

Currently, we are sharing ReplayListenerBus for replaying the event logs of various apps.
https://github.com/apache/spark/blob/v1.4.0/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala#L226

While replaying the event log for a particular app, we add an ApplicationEventListener to the bus.
https://github.com/apache/spark/blob/v1.4.0/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala#L391
But we never remove it.

This results in one ReplayListenerBus being associated with multiple ApplicationEventListener:

15/08/11 00:04:00 log-replay-executor INFO FsHistoryProvider: Replaying log path: hdfs://localhost:9000/spark-history/application_1438993108319_0146_1.snappy
15/08/11 00:04:01 log-replay-executor INFO ApplicationEventListener: onEnvUpdate
15/08/11 00:04:01 log-replay-executor INFO ApplicationEventListener: onApplicationStart
15/08/11 00:04:40 log-replay-executor INFO ApplicationEventListener: onApplicationEnd
15/08/11 00:04:40 log-replay-executor INFO FsHistoryProvider: Elapsed time: 39.730114407s
15/08/11 00:04:40 log-replay-executor INFO FsHistoryProvider: Application log application_1438993108319_0146_1.snappy loaded successfully.
15/08/11 00:04:40 log-replay-executor INFO FsHistoryProvider: Replaying log path: hdfs://localhost:9000/spark-history/application_1438993108319_0126_1.snappy
15/08/11 00:04:40 log-replay-executor INFO ApplicationEventListener: onEnvUpdate
15/08/11 00:04:40 log-replay-executor INFO ApplicationEventListener: onEnvUpdate
15/08/11 00:04:40 log-replay-executor INFO ApplicationEventListener: onApplicationStart
15/08/11 00:04:40 log-replay-executor INFO ApplicationEventListener: onApplicationStart
15/08/11 00:05:00 log-replay-executor INFO ApplicationEventListener: onApplicationEnd
15/08/11 00:05:00 log-replay-executor INFO ApplicationEventListener: onApplicationEnd
15/08/11 00:05:00 log-replay-executor INFO FsHistoryProvider: Elapsed time: 20.483128154s
15/08/11 00:05:00 log-replay-executor INFO FsHistoryProvider: Application log application_1438993108319_0126_1.snappy loaded successfully.
15/08/11 00:05:00 log-replay-executor INFO FsHistoryProvider: Replaying log path: hdfs://localhost:9000/spark-history/application_1438993108319_0116_1.snappy
15/08/11 00:05:00 log-replay-executor INFO ApplicationEventListener: onEnvUpdate
15/08/11 00:05:00 log-replay-executor INFO ApplicationEventListener: onEnvUpdate
15/08/11 00:05:00 log-replay-executor INFO ApplicationEventListener: onEnvUpdate
15/08/11 00:05:00 log-replay-executor INFO ApplicationEventListener: onApplicationStart
15/08/11 00:05:00 log-replay-executor INFO ApplicationEventListener: onApplicationStart
15/08/11 00:05:00 log-replay-executor INFO ApplicationEventListener: onApplicationStart
15/08/11 00:05:29 log-replay-executor INFO ApplicationEventListener: onApplicationEnd
15/08/11 00:05:29 log-replay-executor INFO ApplicationEventListener: onApplicationEnd
15/08/11 00:05:29 log-replay-executor INFO ApplicationEventListener: onApplicationEnd
15/08/11 00:05:29 log-replay-executor INFO FsHistoryProvider: Elapsed time: 29.110070845s
15/08/11 00:05:29 log-replay-executor INFO FsHistoryProvider: Application log application_1438993108319_0116_1.snappy loaded successfully.

We should either remove the listener from the bus or create a new bus for each app.

Issue Links

• links to: [Github] Pull Request #8088 (mindprince)