[SPARK-4598] Paginate stage page to avoid OOM with > 100,000 tasks

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.2.0
• Fix Version/s: 1.5.0
• Component/s: Web UI
• Target Version/s: 1.5.0
• Assignee: Shixiong Zhu
• Reporter: meiyoula
• Created: 25/Nov/14 08:43
• Updated: 07/Aug/15 21:36
• Resolved: 21/Jul/15 16:54

Description

In HistoryServer stage page, clicking the task href in Description, it occurs the GC error. The detail error message is:
2014-11-17 16:36:30,851 | WARN | [qtp1083955615-352] | Error for /history/application_1416206401491_0010/stages/stage/ | org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:590)
java.lang.OutOfMemoryError: GC overhead limit exceeded
2014-11-17 16:36:30,851 | WARN | [qtp1083955615-364] | handle failed | org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:697)
java.lang.OutOfMemoryError: GC overhead limit exceeded

Issue Links

• is duplicated by: SPARK-5899 Viewing specific stage information which contains thousands of tasks will freak out the driver and CPU cores from where it runs
• is duplicated by: SPARK-9037 Task table pagination for the Stage page
• relates to: SPARK-2017 web ui stage page becomes unresponsive when the number of tasks is large
• links to: [Github] Pull Request #3456 (XuTingjun)
• links to: [Github] Pull Request #7399 (zsxwing)