[SPARK-21348] Spark history Server load too slow when eventlog is large

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Duplicate
• Affects Version/s: 2.1.0
• Component/s: Web UI
• Assignee: Unassigned
• Reporter: duyanghao
• Created: 08/Jul/17 02:21
• Updated: 09/Jul/17 02:36
• Resolved: 09/Jul/17 02:33

Description

There is a eventlog file in hdfs as below:
```
rwxrwx-- 3 root supergroup 9634238709 2017-07-08 04:37 hdfs://xxxxx/shared/spark-logs/xxxxx/xxxx.lz4
```

and i start a spark history server to loading that eventlog file,but it is too slow to get spark web ui through that spark history server.

it has requested for several hours but nothing about spark application web ui appears in the browser

And the cpu usage of spark history server is 105.3%
total used-memory in host is 9G

The host has 48 Cores+130G Memory,enough Computing resources for spark history server.

Issue Links

• Blocked: SPARK-18085 SPIP: Better History Server scalability for many / large applications
• duplicates: SPARK-20582 Speed up the restart of HistoryServer using ApplicationAttemptInfo checkpointing
• duplicates: SPARK-18085 SPIP: Better History Server scalability for many / large applications