[DRILL-5270] Improve loading of profiles listing in the WebUI

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.9.0
• Fix Version/s: Future
• Component/s: Web Server
• Assignee: Unassigned
• Reporter: Kunal Khatua
• Reviewer: Arina Ielchiieva
• Created: 17/Feb/17 00:34
• Updated: 22/Jan/20 14:42

Description

Currently, as the number of profiles increase, we reload the same list of profiles from the FS.
An ideal improvement would be to detect if there are any new profiles and only reload from the disk then. Otherwise, a cached list is sufficient.

For a directory of 280K profiles, the load time is close to 6 seconds on a 32 core server. With the caching, we can get it down to as much as a few milliseconds.

To render the cache as invalid, we inspect the last modified time of the directory to confirm whether a reload is needed.

Issue Links

• is related to: DRILL-2362 Drill should manage Query Profiling archiving
• is related to: DRILL-5028 Opening profiles page from web ui gets very slow when a lot of history files have been stored in HDFS or Local FS.
• is related to: DRILL-2861 enhance drill profile file management
• is related to: DRILL-6053 Avoid excessive locking in LocalPersistentStore
• relates to: DRILL-5068 Create a system table for completed profiles - sys.profiles
• links to: GitHub Pull Request #1250
• links to: GitHub Pull Request #1654