[SPARK-11123] Inprove HistoryServer with multithread to relay logs - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Improvement
Status: Resolved
Priority: Major
Resolution: Duplicate
Affects Version/s: None
Fix Version/s: None
Component/s: None
Labels:
None

Description

Now,with Spark 1.4,when I restart HistoryServer,it took over 30 hours to replay over 40 000 log file. What's more,when I have started it,it may take half an hour to relay it and block other logs to be replayed.How about rewrite it with multithread to accelerate replay log.

Attachments

Issue Links

duplicates

SPARK-6951 History server slow startup if the event log directory is large

Resolved

Activity

People

Assignee:: Unassigned

Reporter:: Xie Tingwen

Votes:: 0 Vote for this issue

Watchers:: 1 Start watching this issue

Dates

Created:: 15/Oct/15 04:11

Updated:: 15/Oct/15 22:38

Resolved:: 15/Oct/15 22:38