[SPARK-31196] Server-side processing of History UI list of applications

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Won't Fix
• Affects Version/s: 2.3.0, 2.4.5
• Component/s: Web UI
• Assignee: Unassigned
• Reporter: Pavol Vidlička
• Created: 19/Mar/20 20:58
• Updated: 25/Mar/20 15:48
• Resolved: 25/Mar/20 15:48

Description

Loading the list of applications in the History UI does not scale well with a large number of applications. Fetching and rendering the list for 10k+ applications takes over a minute (much longer for more applications) and tends to freeze the browser.

Using `spark.history.ui.maxApplications` is not a great solution, because (as the name implies), it limits the number of applications shown in the UI, which hinders usability of the History Server.

A solution would be to use server side processing of the DataTable. This would limit amount of data sent to the client and processed by the browser.

This proposed change plays nicely with KVStore abstraction implemented in SPARK-18085, which was supposed to solve some of the scalability issues. It could also definitely solve History UI scalability issues reported for example in SPARK-21254, SPARK-17243, SPARK-17671