[SPARK-1132] Persisting Web UI through refactoring the SparkListener interface

• Type: Improvement
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.9.0
• Fix Version/s: 1.0.0
• Component/s: Spark Core, Web UI
• Assignee: Andrew Or
• Reporter: Andrew Or
• Created: 24/Feb/14 20:46
• Updated: 10/May/15 17:35
• Resolved: 19/Mar/14 13:17

Description

This issue is a spin-off from another issue - https://spark-project.atlassian.net/browse/SPARK-969

The main issue with the existing Spark Web UI is that its information is lost as soon as the application terminates. This is the direct result of the SparkUI being coupled with SparkContext, which is stopped when the application is finished.

The attached document proposes to tackle this by logging SparkListenerEvents to persist information displayed on the Web UI. We take this opportunity to replace the existing format for storing this information, HTML, with one that is more flexible, JSON. This allows further post-hoc analysis of a particular Spark application beyond simply reviving the Web UI.

Issue Links

• blocks: SPARK-1276 Add a Simple History Server for the UI for Yarn / Mesos
• blocks: SPARK-1277 Automatically set the UI persistence directory based on cluster settings
• breaks: SPARK-1417 Spark on Yarn - spark UI link from resourcemanager is broken
• is duplicated by: SPARK-969 Persistent web ui
• is duplicated by: SPARK-1020 SparkListener interfaces should not expose internal types/objects
• is related to: SPARK-1475 Drain event logging queue before stopping event logger
• is related to: SPARK-1407 EventLogging to HDFS doesn't work properly on yarn
• relates to: SPARK-969 Persistent web ui