[SPARK-5522] Accelerate the History Server start

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 1.4.0
• Component/s: Spark Core, Web UI
• Assignee: Liangliang Gu
• Reporter: Liangliang Gu
• Created: 02/Feb/15 05:54
• Updated: 28/Mar/15 15:06
• Resolved: 28/Mar/15 15:06

Description

When starting the history server, all the log files will be fetched and parsed in order to get the applications' meta data e.g. App Name, Start Time, Duration, etc. In our production cluster, there exist 2600 log files (160G) in HDFS and it costs 3 hours to restart the history server, which is a little bit too long for us.

It would be better, if the history server can show logs with missing information during start-up and fill the missing information after fetching and parsing a log file.

Issue Links

• is duplicated by: SPARK-4558 History Server waits ~10s before starting up
• links to: [Github] Pull Request #4525 (marsishandsome)