[SPARK-18085] SPIP: Better History Server scalability for many / large applications

• Type: Umbrella
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 2.3.0
• Component/s: Spark Core, Web UI
• Labels: SPIP
• Assignee: Marcelo Masiero Vanzin
• Reporter: Marcelo Masiero Vanzin
• Shepherd: Marcelo Masiero Vanzin
• Created: 24/Oct/16 23:32
• Updated: 05/Feb/18 16:11
• Resolved: 19/Jan/18 20:45

Description

It's a known fact that the History Server currently has some annoying issues when serving lots of applications, and when serving large applications.

I'm filing this umbrella to track work related to addressing those issues. I'll be attaching a document shortly describing the issues and suggesting a path to how to solve them.

Issue Links

• Blocked: SPARK-21348 Spark history Server load too slow when eventlog is large
• blocks: SPARK-20656 Incremental parsing of event logs in SHS
• is blocked by: SPARK-20205 DAGScheduler posts SparkListenerStageSubmitted before updating stage
• is blocked by: SPARK-20342 DAGScheduler sends SparkListenerTaskEnd before updating task's accumulators
• is duplicated by: SPARK-21348 Spark history Server load too slow when eventlog is large
• is duplicated by: SPARK-20582 Speed up the restart of HistoryServer using ApplicationAttemptInfo checkpointing
• is related to: SPARK-21254 History UI: Taking over 1 minute for initial page display
• is related to: SPARK-19814 Spark History Server Out Of Memory / Extreme GC
• is related to: SPARK-20421 Mark JobProgressListener (and related classes) as deprecated
• relates to: SPARK-6951 History server slow startup if the event log directory is large

Sub-Tasks

• Key-value store abstraction and implementation for storing application data: Resolved
• Use key-value store to keep History Server application listing: Resolved
• Implement listener for saving application status data in key-value store: Resolved
• Hook up Spark UI to the new key-value store backend: Resolved
• Make Environment page use new app state store: Resolved
• Make Executors page use new app state store: Resolved
• Make the Storage page use new app state store: Resolved
• Make Jobs and Stages pages use the new app state store: Resolved
• Simplify REST API class hierarchy: Resolved
• Remove JobProgressListener (and other unneeded classes): Resolved
• Speed up the new app state listener: Resolved
• Make SQL UI use new app state store: Resolved
• Add auto-cleanup of old elements to the new app state store: Resolved
• Add controls for how much disk the SHS can use: Resolved
• In-memory key-value store implementation: Resolved
• Speed up Stage page: Resolved
• Remove stale applications from SHS listing: Resolved
• Move kvstore package to util.kvstore, add annotations: Resolved