Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-9806

Don't share ReplayListenerBus between multiple applications

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.4.0
    • Fix Version/s: 1.4.2, 1.5.0
    • Component/s: Web UI
    • Labels:
      None

      Description

      Currently, we are sharing ReplayListenerBus for replaying the event logs of various apps.
      https://github.com/apache/spark/blob/v1.4.0/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala#L226

      While replaying the event log for a particular app, we add an ApplicationEventListener to the bus.
      https://github.com/apache/spark/blob/v1.4.0/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala#L391
      But we never remove it.

      This results in one ReplayListenerBus being associated with multiple ApplicationEventListener:

      15/08/11 00:04:00 log-replay-executor INFO FsHistoryProvider: Replaying log path: hdfs://localhost:9000/spark-history/application_1438993108319_0146_1.snappy
      15/08/11 00:04:01 log-replay-executor INFO ApplicationEventListener: onEnvUpdate
      15/08/11 00:04:01 log-replay-executor INFO ApplicationEventListener: onApplicationStart
      15/08/11 00:04:40 log-replay-executor INFO ApplicationEventListener: onApplicationEnd
      15/08/11 00:04:40 log-replay-executor INFO FsHistoryProvider: Elapsed time: 39.730114407s
      15/08/11 00:04:40 log-replay-executor INFO FsHistoryProvider: Application log application_1438993108319_0146_1.snappy loaded successfully.
      15/08/11 00:04:40 log-replay-executor INFO FsHistoryProvider: Replaying log path: hdfs://localhost:9000/spark-history/application_1438993108319_0126_1.snappy
      15/08/11 00:04:40 log-replay-executor INFO ApplicationEventListener: onEnvUpdate
      15/08/11 00:04:40 log-replay-executor INFO ApplicationEventListener: onEnvUpdate
      15/08/11 00:04:40 log-replay-executor INFO ApplicationEventListener: onApplicationStart
      15/08/11 00:04:40 log-replay-executor INFO ApplicationEventListener: onApplicationStart
      15/08/11 00:05:00 log-replay-executor INFO ApplicationEventListener: onApplicationEnd
      15/08/11 00:05:00 log-replay-executor INFO ApplicationEventListener: onApplicationEnd
      15/08/11 00:05:00 log-replay-executor INFO FsHistoryProvider: Elapsed time: 20.483128154s
      15/08/11 00:05:00 log-replay-executor INFO FsHistoryProvider: Application log application_1438993108319_0126_1.snappy loaded successfully.
      15/08/11 00:05:00 log-replay-executor INFO FsHistoryProvider: Replaying log path: hdfs://localhost:9000/spark-history/application_1438993108319_0116_1.snappy
      15/08/11 00:05:00 log-replay-executor INFO ApplicationEventListener: onEnvUpdate
      15/08/11 00:05:00 log-replay-executor INFO ApplicationEventListener: onEnvUpdate
      15/08/11 00:05:00 log-replay-executor INFO ApplicationEventListener: onEnvUpdate
      15/08/11 00:05:00 log-replay-executor INFO ApplicationEventListener: onApplicationStart
      15/08/11 00:05:00 log-replay-executor INFO ApplicationEventListener: onApplicationStart
      15/08/11 00:05:00 log-replay-executor INFO ApplicationEventListener: onApplicationStart
      15/08/11 00:05:29 log-replay-executor INFO ApplicationEventListener: onApplicationEnd
      15/08/11 00:05:29 log-replay-executor INFO ApplicationEventListener: onApplicationEnd
      15/08/11 00:05:29 log-replay-executor INFO ApplicationEventListener: onApplicationEnd
      15/08/11 00:05:29 log-replay-executor INFO FsHistoryProvider: Elapsed time: 29.110070845s
      15/08/11 00:05:29 log-replay-executor INFO FsHistoryProvider: Application log application_1438993108319_0116_1.snappy loaded successfully.
      

      We should either remove the listener from the bus or create a new bus for each app.

        Attachments

          Activity

            People

            • Assignee:
              ragarwal Rohit Agarwal
              Reporter:
              ragarwal Rohit Agarwal
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: